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Abstract 

In  1970  the  Department  of  Defense  introduced  the  Department  of  Defense 
Directive  5000  to  standardize  the  acquisition  process;  the  directive  created  oversight 
forums  to  ensure  the  policies  and  procedures  created  were  followed,  track  program 
progress,  and  identify  programs  in  trouble.  Although  oversight  was  essentially  created  to 
help  reduce  the  cost  of  acquisitions,  there  is  reason  to  believe  that  it  may  increase  the 
costs;  however,  because  there  has  only  been  a  few  studies  conducted  that  estimated  the 
cost  of  oversight  no  one  knows  how  much  “oversight”  costs  individual  programs. 
Numerous  oversight  processes  are  being  used  today,  but  no  research  shows  one  process  is 
any  different  from  the  other.  Nor  have  studies  been  done  to  determine  the  cost  drivers 
for  oversight. 

This  thesis  will  provide  a  foundation  and  potential  cost  saving  recommendations 
that  would  benefit  the  Department  of  Defense  in  most  of  the  acquisition  programs  it 
monitors.  An  estimated  cost  of  oversight  will  be  calculated  for  programs  following  three 
different  oversight  processes  using  the  Delphi  Methodology.  The  estimates  will  be 
compared  to  determine  if  there  are  any  statistical  differences  between  them.  A  future 
track  for  the  next  generation  of  oversight  processes  will  develop  from  the 
rec  ommendations . 
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ESTABLISHING  A  FOUNDATION  TO  CAPTURE  THE  COST  OF 


OVERSIGHT  FOR  A  MAJOR  DEFENSE  PROGRAM  WITHIN  THE 
INFORMATION  TECHNOLOGY  (IT)  ACQUISITION  COMMUNITY 

1.0  Introduction 

1.1  Overview 

War  preparation  has  been  one  eonstant  throughout  the  history  of  the  United  States 
(U.S.).  Today  the  U.S.  spends  billions  of  dollars  on  researeh  and  development  and 
proeurement  of  weapon  systems  and  other  major  defense  aequisition  programs  annually. 
For  the  purpose  of  this  thesis,  the  definition  of  major  defense  aequisition  programs 
(MDAP)  will  be  taken  from  the  February  23,  1991  version  of  the  Department  of  Defense 
(DoD)  Instruction  5000.2,  Defense  Acquisition  Management  Policies  and  Procedure. 

That  version  of  the  DoD  Instruction  5000.2  defined  an  MDAP,  as  “a  directed,  funded 
effort  that  is  designed  to  provide  a  new  or  improved  materiel  capability  in  response  to  a 
validated  need.  MDAPs  must  have  an  eventual  cost  of  $200  million  in  research, 
development,  test  and  evaluation  (RDT&E)  cost,  and  at  least  an  eventual  $1  billion  cost 
in  procurement  expenditures  both  of  which  are  in  fiscal  year  (FY)  1980  constant  year 
dollars”  (14:1). 

The  acquisition  cost  for  an  MDAP  can  reach  astounding  numbers.  Therefore,  it  is 
imperative  that  the  stakeholders  in  the  process  be  prudent  and  efficient  when  dealing  with 
these  programs  because  all  funding  for  these  programs  are  appropriated  funds  from 
Congress.  Both  Congress  and  the  DoD  have  a  responsibility  to  the  taxpayers  to  ensure 
the  funds  entrusted  in  them  are  not  caught  up  in  some  type  of  fraud,  waste,  or  abuse. 

Over  the  years,  US  government  agencies  have  tried  to  reduce  the  amount  of  inefficiencies 
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within  the  acquisitions  process  by  developing  and  implementing  policies  and  procedures 
for  the  acquisitions  community.  Official  guidelines  implemented  to  regulate  the 
acquisition  process  date  back  to  the  late  1960s,  and  the  regulations  have  been  changed 
and  reformed  continuously. 

Some  of  the  more  recent  changes  to  the  policies  and  procedures  can  be  attributed 
to  the  fall  of  the  Soviet  Union,  and  that  is  when  the  U.S.’s  approach  to  national  defense 
had  to  change.  After  the  fall  of  the  Soviet  Union,  many  Americans  believed  that  the  U.S. 
could  reduce  spending  on  its  national  defense  programs  because  the  number  one  enemy 
was  no  longer  a  viable  threat.  With  a  decrease  in  support  for  defense  spending,  mainly 
production  dollars  came  budget  cuts  and  a  major  transformation  of  the  DoD.  Major 
advancements  in  technology  also  helped  transform  the  acquisition  process.  Since  the 
defense  budget  shrank  over  the  next  10  years,  the  way  the  DoD  acquired  its  weapon 
systems  and  other  defense  programs  had  to  be  streamlined  to  get  the  most  “bang  for  the 
buck.”  The  government  could  no  longer  afford  to  overrun  budgets  or  delay  schedules 
without  being  accountable  for  each  schedule  delay  or  budget  increase.  Because  the  dollar 
amounts  spent  on  acquiring  MDAPs  is  a  significant  part  of  the  nation’s  defense  spending, 
there  have  been  many  studies  conducted  to  ensure  the  acquisitions  process  is  both 
efficient  and  effective,  although  little,  if  any,  research  has  examined  the  cost  of  oversight. 
While  several  initiatives  have  been  put  in  place  to  reduce  the  cost  of  acquisitions,  there 
has  been  minimal  work  done  on  how  to  capture  the  actual  cost  of  oversight  for  MDAPs. 
Some  programs  have  been  given  the  freedom  to  develop  their  own  policies  and 
procedures,  some  have  received  waivers  from  the  policies  and  procedures  altogether,  and 
some  have  been  given  the  authority  to  use  non-traditional  methods  when  going  through 
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the  approval  process,  but  the  programs  must  use  the  guidance  set  by  the  DoD  5000  series 
which  has  been  put  in  place  to  govern  the  acquisition  process. 

This  thesis  will  estimate  the  actual  cost  of  oversight  of  a  MDAP  that  used  non- 
traditional  methods  for  obtaining  approval  using  the  DoD  5000  series  as  its  governing 
regulations.  This  chapter  will  cover  the  background  and  the  problem,  provide  some 
assumptions,  set  the  scope  and  outline  the  methodology. 

1.2  Background 

Acquisition  oversight  as  we  know  it  today  began  in  the  late  1960s.  “The  Defense 
Systems  Acquisitions  Review  Council  (DSARC)  was  created  to  advise  then  Deputy  of 
Secretary  of  Defense  (SecDef),  David  Packard,  on  the  status  and  readiness  of  each  major 
defense  system  to  proceed  from  one  phase  to  the  next”  (1:20).  The  DSARC  was  used  to 
monitor  the  progress  of  major  acquisitions  programs,  and  would  allow  the  DoD  to 
regulate  the  acquisition  process  and  ensure  that  the  program  was  moving  along  according 
to  schedule.  After  the  creation  of  DSARC,  Packard  authored  two  memorandums.  One 
requested  input  on  ways  to  improve  the  acquisitions  process  and  the  other  cited  ways  the 
acquisitions  process  could  be  improved  (1  ;21).  Packard  believed  that  there  had  to  be 
guidelines  put  in  place  for  all  members  of  the  acquisition  workforce  to  follow.  In  1971 
the  first  set  of  regulations  was  released  and  implemented  DoD  wide.  According  to 
Ferrara,  the  first  copy  of  the  DoD  5000.1  and  DoD  Instruction  5000.2,  better  known  as 
the  DoD  5000  Series  was  issued  in  July  1971  (21:11 1).  It  would  be  one  of  the  first 
attempts  to  regulate  the  acquisition  process.  However,  it  would  only  be  the  beginning  of 
a  string  of  changes  and  reiterations  of  the  same  document.  To  date,  every  presidential 
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administration  since  Nixon’s  administration  has  made  some  type  of  change  to  the  series 
(21:115).  The  latest  update  to  the  series  was  released  in  May  2003. 

1.3  Problem 

During  the  1990s,  aequisition  reform  was  a  major  priority  for  the  Clinton 
administration  as  major  problems  with  the  aequisition  proeess  were  brought  to  publie 
attention.  Many  Americans  felt  the  government  was  abusing  its  power  and  taking 
advantage  of  the  tax  dollars  appropriated  for  aequisitions.  According  to  Gregory,  “in  the 
minds  of  mueh  of  the  citizenry,  the  Pentagon  proeurement  system  is  seandalous  and  the 
defense  industry  is  manned  by  fast-buck  artists,  incompetents,  or  deranged  Dr. 
Strangeloves  who,  when  they  lack  weapons  of  mass  destruetion  to  tinker  with,  design 
$600  hammers  or  $5,000  eoffeepots”  (24:1). 

Since  most  Amerieans  believed  paying  $600  for  hammers  or  $5,000  for 
eoffeepots  was  excessive,  DoD  looked  at  the  way  they  aequired  items.  The  coffeepot  and 
hammer  eost  were  both  exaggerated  examples  of  the  publie  perception;  however,  these 
incidents  were  “wake  up”  calls  to  the  government  along  with  the  aequisitions  eommunity. 
Because  the  government’s  aequisition  process  is  subject  to  public  scrutiny.  Congress  and 
DoD  must  ensure  that  the  policies  and  proeedures  put  in  place  to  regulate  the  proeess  are 
flexible  yet  stringent  enough  to  react  to  the  forever  changing  needs  of  the  government. 
Although  oversight  is  needed  within  the  process,  it  must  be  tailored  to  the  situation  in 
order  to  eliminate  non-value-added  proeesses  that  only  add  to  the  increasingly  high  cost 
of  acquisitions. 
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1.4  Assumptions 


For  the  purpose  of  produeing  results  that  ean  be  eompared  to  the  results  gathered 
from  other  MDAP  programs  within  traditional  and  space  environment,  some  assumptions 
had  to  be  made.  The  following  assumptions  were  used  for  the  purpose  of  conducting  this 
research; 


1 .  Assume  that  the  program  has  stable  requirements  and  a  stable  budget 

2.  Assume  that  the  studies  performed  on  the  oversight  process  accurately 
depict  the  normal  process  as  defined  in  the  DoD  5000  series 

3.  Assume  the  exterior  agencies  or  services  do  not  adversely  affect  the 
measures  of  performance  used  by  the  programs  being  studied 

4.  Assume  the  cost  are  associated  only  with  parameters  above  the 
Program  Executive  Officer  (PEO)  level 

5.  Assume  that  the  oversight  cost  collected  occurs  post  Milestone  A 
approval 

The  assumptions  will  help  increase  the  interpretability  of  the  data  observed.  By 
narrowing  the  scope  of  the  study,  the  results  of  this  research  can  be  compared  to  the 
research  being  conducted  for  oversight  cost  in  Space  and  normal  DoD  5000  Series 
programs. 


1.5  Scope 

“In  2001,  the  Eindersecretary  of  Acquisition  Technology  and  Eogistics  [USD 
(AT&E)],  along  with  the  Assistant  Secretary  of  Defense  (Communications,  Command, 
Control,  and  Intelligence)  [ASD  (C3I)],  tasked  a  Rapid  Improvement  Team  (RIT)  to 
address  reducing  the  time  it  took  to  deliver  mission  effective  Information  Technology 
(IT)  capabilities  within  18  months  or  less  to  the  warfighter”  (29:  slide  2).  According  to 
the  briefing,  MDAP  programs  from  the  Army,  Navy,  and  the  Air  Eorce  participated  in  the 
RIT  pilot  study  (29:  slide  5);  however,  this  thesis  focuses  on  Air  Eorce  test  programs 
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completed  the  pilot  study.  The  RIT  pilot  study  was  an  attempt  to  reduee  aequisition  eyele 
time  by  using  eomputer  teehnology  and  other  innovative  ideas  throughout  the  approval 
proeess.  Although  the  polieies  and  proeedures  set  by  the  DoD  5000  series  were  used,  the 
RIT  established  a  virtual  proeess  that  helped  reduee  the  amount  of  non  value  added 
meetings  required  throughout  the  proeess  by  putting  all  available  data  at  eaeh  partieipants 
fingertips.  The  RIT  also  foeused  on  ereating  a  risk-based  approaeh  to  oversight  that 
allows  the  amount  of  oversight  for  a  partieular  program  be  tailored  to  fit  the  need  based 
on  the  risk  levels  of  that  particular  program  (29:  slide  9).  The  results  of  the  pilot  study 
may  eventually  lead  to  ehanges  in  the  way  oversight  of  MDAPs  are  implemented.  The 
results  of  this  researeh  will  provide  senior  leaders  a  first  look  at  just  how  mueh  oversight 
of  this  type  of  program  eost,  and  it  will  help  them  make  better  deeisions  on  just  how 
mueh  oversight  is  neeessary  for  eaeh  individual  program. 

1.6  Methodology 

The  first  step  in  eondueting  this  researeh  was  to  seleet  an  MDAP  that  was  a  part 
of  the  RIT  pilot  study,  and  gain  support  from  individuals  within  the  IT  eommunity.  By 
reeeiving  the  support  from  these  individuals  early,  it  ensures  that  the  data  required  to 
eonduet  this  study  is  available  to  obtain  and  analyze.  No  partieular  program  will  be 
studied;  however,  individuals  who  are  experts  in  the  field  will  be  utilized  to  gain  enough 
insight  to  answer  the  following  questions. 

1 .  How  many  meetings  were  there? 

2.  Who  attended  those  meetings? 

3.  How  long  did  they  last? 

4.  Were  the  meetings  loeal  or  was  there  temporary  duty  (TDY)  involved? 

5.  How  mueh  time  was  used  in  preparation  of  the  meeting? 
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The  Delphi  method  will  be  used  to  generate  the  overall  program  oversight  cost  estimate. 
A  series  of  questions  will  be  formulated  and  distributed  to  a  panel  of  experts  within  the 
Information  Technology  (IT)  acquisitions  community.  The  expert  panel  will  be  asked  to 
answer  the  questions  and  return  them  to  be  analyzed.  After  validation  of  the  data  is 
complete,  cost  per  hour  rates  will  be  calculated  for  each  individual  within  the  process 
how  has  vital  role  in  getting  a  program  approved  through  the  acquisitions  process.  Some 
of  the  individuals  included  would  be  the  members  of  the  integrated  program  team  (IPT) 
associated  with  IT  programs  that  took  part  in  the  RIT  pilot  study.  Members  will  be 
recommended  by  their  peers  and  colleagues  but  will  remain  anonymous  to  one  another. 

In  the  final  stage,  the  total  cost  for  TDYs,  meetings,  and  personnel  will  be  added  together 
to  calculate  the  total  cost  of  oversight  for  a  MDAP  for  one  Milestone  Decision  Point 
(MDP).  One  important  thing  to  remember  is  that  this  research  focuses  on  Air  Force 
programs  only. 

1.7  Research  Objectives  and  Questions 

There  have  been  many  reforms  initiated  to  help  reduce  the  cost  of  oversight 
within  the  information  technology,  missile  defense,  and  the  space  and  missile 
acquisition  communities.  However,  no  one  has  done  any  research  to  determine  if 
these  reforms  were  more  effective.  Therefore,  the  main  objective  of  this  research  is 
to  determine  the  cost  of  oversight  within  the  information  technology  community  after 
using  a  new  “virtual”  approval  process  to  help  reduce  the  amount  of  time  to  obtain 
Milestone  Decision  Authority  (MDA)  approval.  The  second  objective  is  to  compare 
the  results  with  other  MDAPs  (Normal  DoD  5000  series  rules  and  regulations  and 
Space  and  Missiles)  using  different  processes  to  obtain  MDA  approval.  This 
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comparison  will  help  determine  if  there  is  a  superior  process  or  it  will  show  there  are 
no  real  differences  between  the  different  proeesses.  The  researeh  will  also  determine 
if  there  are  specific  cost  drivers  for  oversight.  If  the  cost  drivers  are  determined,  it 
will  help  members  of  the  aequisition  community  better  focus  their  efforts  to  make  the 
process  more  efficient. 

1.8  Summary 

This  ehapter  introduced  the  reason  for  this  research.  It  also  outlined  the  seope 
and  gave  a  brief  introduetion  to  the  methodology  that  will  be  followed  for  this 
research.  Chapter  2  of  the  thesis  will  give  provide  information  related  to  the  research 
from  other  sources  and  will  give  a  historical  picture  of  how  aequisition  has  evolved 
over  the  years,  and  it  will  also  trace  the  reforms  of  the  DoD  5000  series.  More 
information  on  the  Delphi  Method  will  be  revealed  in  Chapter  3.  Chapter  3  defines  a 
speeific  eourse  of  aetion  to  be  followed  throughout  the  research  process.  In  Chapter  4, 
the  results  of  the  research  will  be  given  in  detail.  It  will  analyze  the  data  eollected 
using  the  proeedures  set  in  Chapter  3.  Conelusions,  reeommendations,  and  potential 
follow  up  researeh  will  be  covered  in  the  final  chapter  of  this  researeh. 
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2.0  Literature  Review 


2.1  Overview 

Over  the  years,  many  studies  have  been  eonducted  on  the  Department  of  Defense 
(DoD)  aequisition  proeess.  After  all,  the  United  States  (U.S)  government  spends  billions 
of  dollars  on  weapon  systems,  information  teehnology,  and  space  programs  annually. 
According  to  an  audit  report  filed  by  the  Air  Force  Inspector  General’s  office  numbered 
92-047,  “as  of  March  1991,  DoD  had  approximately  100  active  major  acquisition 
programs  with  a  total  value  of  $838  billion”  (14:1).  In  another  report  authored  by 
Czelusniak  and  Rodgers,  they  reported  that  “of  the  billions  of  dollars  spent  on  the 
nation’s  defense,  DoD  loses  approximately  $5  billion  per  year  in  investment  funds  due  to 
cost  growth”  (10:16).  Because  appropriated  funds  are  used  for  the  acquisition  process,  it 
is  imperative  for  the  DoD  to  utilize  these  funds  effectively  and  efficiently.  Although 
DoD  has  continued  to  transform  the  acquisition  process  to  meet  the  needs  of  our 
government,  there  is  still  room  for  improvement. 

In  an  attempt  to  keep  cost  growth  from  becoming  too  expensive,  our  leaders 
continuously  monitor  cost  growth  within  our  acquisition  programs  and  in  the  early  1970s, 
the  government  released  formal  policies  and  procedures  for  the  acquisition  process. 
According  to  Ferrara,  “the  first  versions  of  DoD  Directive  5000.1(DoDD  5000.1)  and 
DoD  Instruction  5000.2  (DoDI  5000.2),  which  are  also  known  as  the  DoD  5000  series, 
were  released  in  mid  1970  and  1971”  (21:109).  The  creation  of  the  DoD  5000  series  was 
the  first  step  by  the  government  to  stabilize  the  acquisitions  process.  Along  with  the  DoD 
5000  series,  oversight  groups  and  forums  were  also  created  to  help  oversee  the  programs 
going  through  the  acquisitions  process.  In  a  report  to  the  Secretary  of  Defense  (SecDef) 
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in  1994,  it  stated  that  “oversight  verifies,  on  a  continuing  basis,  that  the  program  is 
proceeding  according  to  expectations”  (5:15).  The  DoD  5000  series  can  be  considered 
the  “bible”  for  the  acquisition  process;  however,  it  is  a  bible  that  constantly  changes. 
Over  the  last  20  years,  there  have  been  many  revisions  and  editions  of  the  DoD  5000 
series,  and  the  oversight  groups  and  forums  have  come  and  gone.  One  thing  has 
remained  constant  throughout  the  years  and  that  is  change.  However,  changes  to  the 
groups  and  forums  as  well  as  the  5000  series  will  continue  as  DoD  and  our  federal 
government  tries  to  find  a  balance  between  the  flexibility  within  a  program  office  and 
control  of  cost  overruns. 

The  first  section  of  this  chapter  will  identify  and  define  the  key  terms  used  in 
throughout  this  thesis.  This  chapter  will  also  briefly  discuss  how  typical  MDAP  operates 
as  outlined  in  the  DoD  5000  series.  An  evolution  of  the  DoD  5000  series  and  the 
importance  of  it  will  follow  that  section  to  give  a  better  understanding  of  why  changes 
are  ongoing.  Presidential  administration  influences  will  be  discussed,  and  the  final 
section  of  the  chapter  will  discuss  previous  research  in  the  area  of  cost  oversight. 

2.2  Definitions 

In  order  to  lay  a  framework  for  discussion  some  key  terms  must  be  identified  and 
defined.  To  begin,  for  the  purpose  of  this  thesis  oversight  is  defined  as  “the  cost 
associated  with  the  approval  process  of  an  acquisition  program  for  each  Milestone 
Decision  Authority  (MDA)  above  the  Program  Manager  (PM)  level.”  According  to  the 
DoDl  5000.2,  “a  PM  is  designated  for  each  acquisition  program  no  later  than  program 
initiation”  (15:35).  The  PM  normally  reports  to  a  Program  Executive  Officer  (PEO). 

The  DoD  5000  series  says  that  “unless  a  waiver  is  granted,  a  PEO  has  to  be  assigned  to 
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all  Acquisition  Category  I  (ACAT  I),  Acquisition  Category  lA  (ACAT  lA),  and  sensitive 
classified  programs,  or  for  any  other  program  determined  by  the  Component  Acquisition 
Executive  (CAE)  to  require  dedicated  executive  management”  (15:35).  The  PEO  is  the 
executive  manager  over  the  PM  or  PMs  they  are  assigned  and  report  to  the  Defense 
Acquisitions  Board  (DAB).  The  DAB  is  the  final  formal  member  of  the  vertical  chain  of 
command.  And  the  MDA  is  the  final  approval  authority  in  determining  whether  a 
program  moves  horizontally  from  one  milestone  to  the  next.  Eor  the  purpose  of  this 
thesis,  only  the  vertical  levels  of  approval  above  the  PM  will  be  considered  when 
determining  the  cost  of  oversight.  In  figure  2.1  below,  the  vertical  levels  of  approval  are 
shown  in  yellow. 


Eigure  2.1:  Oversight  Approval  Eevels 

The  vertical  levels  of  hierarchy  in  figure  2.1  appear  to  be  streamlined.  However, 
outside  influences  are  not  accounted  for  in  this  diagram.  Although  the  graph  doesn’t 
show  it,  the  Working  Integrated  Product  Team  (WIPT)  works  under  the  Overarching 
Integrated  Product  Team  (OIPT)  and  neither  one  is  a  member  of  the  formal  chain  of 
command  for  the  program  approval  process,  they  can  have  a  major  impact  on  a 
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program’s  progress.  An  Integrated  Product  Team  (IPX)  is  a  group  of  functional  experts 
that  has  a  stake  in  the  operation  and  success  of  their  assigned  program.  By  using  this 
type  of  collaboration  method,  programs  hope  to  eliminate  “stovepipe”  holdups  within  a 
program.  Since  teams  are  made  up  of  functional  experts,  they  are  able  to  identity  and 
solve  problems  quickly  to  keep  their  program  on  schedule.  According  to  Engel,  IPTs 
were  formally  codified  in  the  March  1996  version  of  the  DoD  5000  series  (19:25).  In 
chapter  4,  these  levels  will  be  dissected  further  to  capture  the  number  meetings  necessary 
before  a  program  is  ready  to  go  to  the  DAB  to  be  reviewed  for  milestone  approval.  It 
will  also  place  a  cost  of  oversight  required  to  obtain  an  MDA  approval. 

2.3  MDAP  Operation 

Since  the  early  1970s,  the  MDAP  process  has  been  governed  by  the  DoD  5000 
series.  Lead  by  then  SecDef  David  Packard,  DoD  set  out  to  create  some  boundaries  for 
acquisition  officials  that  would  lead  to  a  reduction  in  the  cost  of  acquisitions.  And  over 
the  last  20  years  the  ideas  set  by  Packard  have  not  changed,  but  the  acquisition’s  process 
has  changed  numerous  times  since  the  creation  of  the  DoD  5000  series.  This  research 
will  evaluate  how  a  waiver  of  procedures  to  the  DoD  5000  series  has  impacted  the  cost  of 
oversight  for  the  IT  community.  Although  the  IT  community  must  abide  by  the  rules  and 
regulations  of  the  DoD  5000  series,  it  has  been  given  a  waiver  on  the  “process”.  This 
thesis  will  determine  if  the  changes  made  to  the  process  increases  or  decreases  the  cost  of 
oversight  compared  to  that  of  a  program  not  granted  a  waiver. 

2,3.1  Development  Stages 

Other  than  making  progress  through  the  vertical  chain  of  command,  a  program 
must  also  make  progress  through  a  horizontal  development  cycle.  Although  this  research 
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focuses  on  a  specific  section  of  the  development  cycle,  figure  2.2  shows  the  horizontal 
steps  a  MDAP  must  progress  through  for  the  normal  DoD  5000  series  process. 


Process  entry  at  Milestones  A,  B,  or  C 
Entrance  criteria  met  before  entering  phase 

Evolutionary  Acquisition  or  Single  Step  to  Full 
Capability 


(Program 

Initiation) 


IOC 


FOC 


Concept 

Technology 

System  Development 

Production  & 

Operations  & 

Refinement 

Development 

&  Demonstration 

Deployment 

Support 

\  Concept 
/  Decision 

Design 

C  >  Readiness 
Review 

LRIP/IOT&E  ^  D^fision 

Review 

Pre-Systems  Acquisition 


Systems  Acquisition 


Sustainment 


Figure  2.2:  Forward  Progress  (15:2) 

The  systems  acquisition  process  is  composed  of  all  the  steps  included  within  Milestone  B 
and  Milestone  C.  There  are  steps  under  Milestone  C  that  continue  on  after  a  TRIP 
decision,  but  these  are  outside  the  scope  of  this  research.  It  focuses  on  getting  approval 
to  progress  from  Milestone  B,  the  System  Development  and  Demonstration  (SDD)  stage, 
and  enter  Milestone  C,  the  Production  and  Deployment  (PD)  stage.  “The  purpose  of  the 
SDD  stage  is  to  develop  a  system  or  an  increment,  capability;  reduce  integration  and 
manufacturing  risk. . .”  (15:6).  During  Milestone  A,  ideas  and  needs  are  refined,  but  the 
program  does  not  enter  Milestone  B  (program  initiation)  until  it  has  been  approved.  And 
at  that  point,  the  program  is  officially  established. 

Once  the  program  enters  the  SDD  stage,  “the  PM  must  prepare  an  acquisition 
strategy  to  guide  the  activities  during  this  stage  and  it  must  be  approved  by  the  MDA” 
(15:7).  There  are  two  parts  to  the  SDD:  System  Integration  and  System  Demonstration. 
The  DoDI  5000.2  says  that  “there  will  be  a  Design  Readiness  Review  to  provide  a  mid¬ 
phase  assessment  of  design  maturity  as  evidenced  by  the  number  of  subsystems  and 
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design  reviews  eompleted  sueeessfully”  (15:8).  In  order  to  exit  the  SDD  stage,  Systems 
Integration  and  Demonstration  have  to  be  proven,  and  the  MDA  must  approve  eaeh 
before  the  program  ean  progress  to  the  following  stage.  At  the  end  of  the  SDD  a  deeision 
will  be  made  to  either  end  the  program  or  approve  the  program  for  Milestone  C  and 
authorizes  it  to  begin  Low-Rate  Initial  Produetion  (LRIP). 

The  purpose  of  LRIP  is  to  “result  in  eompletion  of  manufaeturing  development  in 
order  to  ensure  adequate  and  efficient  manufacturing  capability  and  to  produce  the 
minimum  quantity  necessary  to  provide  production  or  production  representative  articles 
for  . .  .<testing>”  (15:9).  During  LRIP  production  quantities  are  limited  in  order  to 
provide  enough  units  to  ensure  testing  and  producibility  yet  limit  the  taxpayer  expense 
before  fully  approving  production.  According  to  DoDI  5000.2,  the  quantity  is  normally 
limited  to,  “10  percent  of  the  total  production  quantity  documented  in  the  acquisition 
strategy”  (15:9).  Once  LRIP  has  proven  to  be  successful,  the  MDA  can  approve  the 
program  for  Full-Rate  Production  (FRIP)  and  delivery  of  the  systems  to  the  field.  These 
systems  must  attain  Initial  Operational  Capability.  The  statutory  and  regulatory 
requirements  are  found  in  tables  at  Enclosure  3  of  the  DoDI  5000.2  (15:10).  Once  the 
systems  are  fielded  to  the  war  fighters  the  program  progresses  to  the  Operations  and 
Support  stage  and  after  the  system  is  relieved  of  duty  it  moves  into  the  Disposal  phase 
which  officially  ends  the  program. 

2,3.2  Differences  in  IT  Development 

Although  the  cycles  for  the  IT  community  resemble  the  normal  DoD  5000  series 
development  cycles,  they  have  created  their  own  process.  The  IT  community  needed  to 
focus  on  reducing  the  time  needed  to  field  a  system.  Therefore,  the  RIT  began  to  assign 
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oversight  based  on  risk.  The  needs  of  the  IT  community  were  different  from  that  of  the 
Space  and  normal  DoD  5000  Series  programs  because  they  each  had  different  life  cycles. 
These  life  cycles  made  it  difficult  to  have  just  one  way  of  doing  acquisition.  The 
different  life  cycles  are  illustrated  below  in  Figure  2.3. 


Life  Cycle  Profiles 


Figure  2.3;  Life  Cycle  Profdes  (2;  slide  5) 

The  typical  DoD  5000  Series,  IT  and  Space  programs  require  efficient  acquisition 
processes  and  the  government  is  trying  to  adapt  to  the  different  situations  its  faced  with 
over  time.  More  on  the  evolution  of  the  DoD  5000  Series  and  the  acquisition  process 
will  be  discussed  in  the  following  section. 

2.4  Evolution  of  the  DoD  5000  Series  of  Instructions 

Now  that  the  acquisition  processes  for  both  the  normal  DoD  5000  series  program 
and  that  of  the  IT  community  have  been  described,  the  next  step  is  to  discuss  the 
evolution  of  the  DoD  5000  series.  Since  its  inception  in  the  early  1970s,  the  DoD  5000 
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series  has  served  as  guidanee  for  the  aequisition  eommunity.  Joe  Ferrara  said  in  a  1996 
artiele  that,  the  “DoDD  5000.1  and  DoDl  5000.2  have  been  the  foundation  for  the 
defense  aequisition  proeess  for  over  20  years”  (21 ;  109).  Sinee  the  issuance  of  the  first 
versions  of  the  series,  the  DoD  5000  series  has  been  rewritten  over  10  times,  with  the  first 
nine  issuances  between  1971  and  1993  according  to  Ferrara  (21:109).  If  studied,  each 
update  to  the  series  provides  a  bit  of  historical  evidence  of  the  political  and  economical 
climate.  The  DoD  5000  series  was  usually  updated  as  a  reaction  to  either  the  changing  of 
presidential  administrations  or  changes  within  the  economy.  Ferrara  points  out  that  the 
reason  it  is  important  to  study  the  evolution  of  the  DoD  5000  series  is  that,  “the  5000 
documents  offer  a  unique  window  on  the  evolution  of  policy  in  a  major  government 
department”  (21:109).  Because  the  DoD  5000  series  is  the  “bible”  for  the  acquisition 
process,  any  revisions  to  it  served  as  an  effort  to  reform  the  acquisition  process  to  better 
suit  the  needs  of  the  government  and  the  way  it  conducted  business. 

2.4,1  Secretary  Packard  Leads  the  Way 

Deputy  Secretary  of  Defense,  David  Packard,  in  the  late  1960s  began  to  feel 
uneasy  about  the  direction  of  the  acquisition  of  defense  programs.  He  believed  that  the 
defense  acquisition  process  could  use  some  improvements  (21:111).  Packard  also 
recognized  that  DoD  needed  to  more  effectively  manage  acquisition  and  control  cost 
growth  (21:110).  Deputy  Secretary  Packard  headed  a  defense  acquisition  review  council 
charged  to  examine  the  defense  acquisition  process  to  discover  opportunities  to  improve 
the  process.  In  May  1970,  Packard  issued  a  memorandum  in  which  the  DoD  5000  series 
was  conceived  (21:111).  This  memorandum  outlined  the  ideas  that  would  later  form  the 
basis  for  the  first  issuance  of  DoD  Directive  5000.1.  Some  of  Packard’s  ideas  listed 
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included,  “decentralized  execution,  streamlined  management  structures,  and  use  of 
appropriate  contract  mechanisms”  (21:111).  The  first  edition  of  the  Department  of 
Defense  Instruction  5000.1  (DoDI  5000.1)  was  issued  in  July  1971  and  it  was  fdled  with 
the  original  ideas  of  Packard  (21:11 1).  Ferrara  suggests  the  original  guidelines  for  the 
operation  of  a  defense  acquisition  program  as  outlined  in  the  5000.1  (as  envisioned  by 
Deputy  Secretary  Packard)  have  been  the  driving  force  behind  every  acquisition  reform 
effort  and  DoD  5000  revision  ever  since  (21:111).  There  will  be  changes  made  by  each 
presidential  administration  to  the  original  instructions.  Although  the  first  edition  stated 
that  “Layers  of  authority  between  the  program  manager  and  his  Component  Head  shall  be 
kept  to  a  minimum”  (21:111),  senior  leaders  continue  to  struggle  with  that  objective. 
Advances  in  technology  have  forced  DoD  to  adapt  to  a  new  economical  and  acquisition 
environment.  It  is  assumed  that  Packard  had  an  idea  that  the  forces  of  society  would  lead 
to  process  changes;  therefore,  he  wanted  to  reduce  the  amount  of  hassle  and  unnecessary 
red  tape  to  help  a  program  continue  to  progress  through  the  process.  However,  with  each 
new  publication  of  the  DoD  5000  Series,  there  is  an  attempt  to  find  the  balance  of 
oversight  that  provides  the  greatest  amount  of  flexibility  and  ensures  the  most  efficient 
deployment  of  products  to  the  warfighter. 

2,4,2  Consistent  Themes  of  the  DoD  5000  Series 

Over  the  different  variation  of  the  DoD  5000  Series  there  have  been  some 
consistent  principles  that  remained  the  same  throughout.  Ferrara  suggests  that  DoD  has 
consistently  stressed  the  importance  of  centralized  policy-making  and  decentralized 
program  execution,  fly  before  buy,  streamlined  organizations,  limited  reporting 
requirements,  and  program  stability  (21 : 113-115).  By  having  a  central  board  develop 
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policy,  the  DoD  acquisition  policies  will  have  the  same  focus.  It  also  ensures  that  each 
program  has  a  standard  set  of  guidelines  to  follow.  By  doing  so,  DoD  hopes  that  these 
policies  would  help  keep  the  cost  of  acquisition  to  a  minimum.  Because  the  policy 
makers  have  the  best  interest  of  the  Department  in  mind  and  not  their  program,  they  are 
better  suited  to  make  policies  for  all  DoD  entities.  However,  they  realized  that  the  day- 
to-day  decisions  of  a  program  should  be  made  by  someone  who  has  access  to  the  inside 
data  on  that  program.  These  individuals  know  what  is  best  for  that  program  and  can 
make  decisions  that  will  help  the  program  along  the  different  stages  of  development. 

They  have  a  better  idea  of  what  tradeoffs  need  to  be  made  when  issues  arise.  This  type  of 
strategy  is  known  today  as  “tailoring”  or  “streamlining”  the  acquisition  process  to  fit  a 
particular  program’s  requirements.  The  basis  of  thesis  was  derived  on  the  effects 
tailoring  or  streamlining  has  on  a  program’s  oversight  cost  (21:1 14).  Fly  before  you  buy 
stresses  the  importance  of  testing.  If  taxpayer  dollars  are  being  spent,  there  should  be 
some  proof  that  the  dollars  are  being  spent  on  a  product  that  does  the  job  the  government 
needs  it  to  do.  Each  edition  specifies  that  dollars  should  not  be  committed  until  it  first 
proves  useful  to  the  warfighter  and  producible  given  the  current  industrial  base  and 
technology.  Prototypes  should  be  used  when  necessary  and  the  product  should  be 
thoroughly  tested  prior  to  system  fielding.  Streamlined  organizations  are  stressed  in  each 
revision  because  it  is  important  not  to  have  excessive  layers  in  the  acquisition  process. 
Extra  layers  are  more  detrimental  to  an  efficient  operation  because  they  slow  down  the 
process  and  add  additional  costs.  Eimited  reporting  requirements  attempts  to  remove 
duplicated  efforts.  Eerrara  called  these  themes  the  “management  principles  etched  in  the 
granite  of  the  [first]  5000.1”  (21:113)  and  supports  this  observation  in  several  instances 
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by  comparing  how  different  revisions  of  DoDI  5000.1  address  and  incorporate  the  key 
themes  in  a  similar  manner. 

2,4,3  Changes  by  Administration 

As  stated  before,  the  DoD  5000  Series  has  had  many  changes  sinee  its  inception 
in  the  early  1970s.  One  of  the  major  drivers  for  the  changes  has  been  presidential 
influenee.  Beginning  with  the  Nixon  era,  eaeh  administration  has  ehanged  or  revised  the 
DoD  5000  Series  in  some  way.  Aecording  to  Ferrara,  the  DoDl  5000.1  was  first  issued 
in  1971  under  Nixon  with  two  revisions  under  President  Ford  (1975,1977);  one  revision 
under  Carter  (1980);  four  revisions  under  Reagan(  1982, 1985, 1986, 1987);  one  revision 
under  Bush(1991)  and  one  revision  under  Clinton(1993)  (21:115).  There  have  also  been 
two  revisions  under  the  current  President  Bush  (2000,  2003).  The  following  seetions  will 
try  to  highlight  the  changes  made  during  each  administration. 

2,4,3, 1  The  Nixon  Administration  (1968-1974) 

The  eonception  of  the  DoD  5000  Series  was  a  reaction  to  the  rising  cost  of 
defense  acquisition  costs.  It  was  under  President  Nixon’s  administration  that  the  first 
edition  of  the  DoDl  5000. 1  was  released.  It  was  the  first  official  policy  set  by  the 
government.  Created  from  the  ideas  of  Paekard,  it  outlined  both  the  vertieal  layers  of 
hierarehy  and  the  horizontal  steps  required  by  a  program  in  order  to  reaeh  full  production. 
Under  the  first  series,  the  horizontal  steps  included  program  initiation,  full  scale 
development,  and  produetion/deployment  (21:112).  These  steps  are  somewhat  similar  to 
today’s  milestones  A  through  C  steps,  but  the  final  approval  authority  in  the  vertical  layer 
that  enabled  a  program  to  move  from  one  milestone  to  the  next  went  all  the  way  to  the 
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Secretary  of  Defense  level.  Nixon’s  administration  is  credited  for  setting  the  first  official 
rules  and  regulations  governing  the  DoD  acquisition  process. 

2.4.3.2  The  Ford  Administration  (1974-1977) 

Under  the  Ford  Administration,  there  were  two  revisions  made  to  the  DoD  5000 
Series.  The  first  change  by  his  administration  was  made  in  1975.  Few  changes  were 
made  to  the  document;  however,  the  major  change  was  the  issuance  of  an  accompanying 
instruction,  DoDI  5000.2  signed  by  the  then-Director  of  Defense  Research  and 
Engineering  (21:116).  The  changes  made  the  new  document  more  user-friendly  and  it 
brought  a  greater  focus  to  the  series  itself. 

The  1977  revision  came  in  response  to  “the  recommendations  of  the  commission 
on  government,  the  establishment  of  the  Office  of  Federal  Procurement  Policy,  and  the 
issuance  of  Office  of  Management  and  Budget  Circular  A- 109,  and  it  instituted  a  new 
milestone  decision  point;  Demonstration  and  Validation  (21:117).  This  initiative 
attempted  to  mitigate  technical  risks  as  early  as  possible  in  the  life  of  a  program.  Ferrara 
asserts  that  this  event  was  likely  brought  about  in  part  due  to  the  large  amount  of  money 
being  spent  to  keep  up  with  the  Russians  as  the  Cold  War  had  not  yet  ended  (21:117). 
2,4,3,3,  The  Carter  Administration  (1977-1981) 

The  Carter  administration’s  revision  in  1980  was  very  aggressive.  It  attempted  to 
reduce  cycle  time  in  order  to  get  products  to  the  warfighter  more  quickly  and  add  more 
detail  in  the  form  of  requiring  new  documents.  In  support  of  reducing  cycle  time,  this 
version  authorized  services  to  do  some  novel  things  including,  “omitting  phases 
altogether”  (21:118).  This  version  also  expanded  the  descriptive  nature  of  the  DoD 

5000.2  Instruction,  added  a  new  document,  the  Integrated  Program  Summary  (IPS)  to  the 
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list  of  reports  required  at  a  major  milestone  review.  Aeeording  to  the  revision,  the 
purpose  of  the  IPS  was  to  provide  a  doeument  in  whieh  the  serviee  in  eharge  of  the 
program  could  summarize  the  implementation  plan  for  the  life  cycle  of  the  product  being 
developed  (21:118).  The  position  of  the  DSARC  Executive  Secretary  was  also  described 
in  the  new  version. 

2.43.4  The  Reagan  Administration  (1981-1989) 

The  Reagan  administration  has  to  be  considered  the  most  active  administration  for 
making  changes  to  the  DoD  5000  Series  because  there  were  four  revisions  during  his 
eight  year  administration.  The  administration  released  its  first  DoD  5000  Series  revision 
in  1985.  The  series  was  revised  in  response  to  the  acquisition  horror  stories  about  that 
made  its  way  to  the  public.  Accusations  of  $900  hammers  and  $500  toilet  seats  plagued 
the  media  and  the  government  had  to  respond.  Wilbur  Jones  describes  how  these  stories 
affected  the  climate  in  Congress  in  his  book  Arming  the  Eagle,  “Congress  at  mid-decade 
was  overloaded  with  some  150  different  defense  procurement  bills  in  the  hopper,  many 
counter  productive  and  contradictory”  (27:374).  As  a  response  to  Congress,  the  1985 
version  created  the  Defense  Acquisition  Executive  (DAE).  The  DAE  became  the  single 
accountable  point  of  contact  over  the  approval  of  each  acquisition  program  (21:119). 

Great  change  was  in  store  for  the  DoD  5000  Series  and  the  defense  acquisitions 
system  between  1986  and  1987.  Congress  enacted  the  Defense  Acquisition  Improvement 
Act  in  1986  to  implement  the  Packard  Commission  recommendations  (21:20).  The  act 
created  the  Under  Secretary  of  Defense  for  Acquisition  (USD(A))  in  response  to 
recommendation  made  by  the  Packard  Commission  that  was  formed  in  1985  (21:120). 
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Almost  immediately  after  assuming  the  role  of  USD  (A),  Richard  Godwin 
initiated  another  version  of  the  DoD  5000  series.  In  the  new  version,  it  codified  the  new 
streamlined  the  chain  of  command  for  the  acquisition  process;  it  ran  from  the  PM  thru  the 
PEO  to  the  Acquisition  Executive.  Previously,  the  Secretary  of  Defense  held  the  role  of 
the  acquisition  executive  and  corresponding  role  of  milestone  decision  authority.  The 
new  edition  also  created  three  committees  that  were  focused  on  programmatic  matters: 
strategic  systems,  conventional  systems,  and  C3I  systems  (21:120).  The  reason  for  this 
was  to  streamline  and  cut  down  on  the  number  of  committees  that  met  with  the  new 
Under  Secretary  as  the  chair  of  the  DAB  and  MDA.  The  article  states  that  at  “one  count 
[the  number  of  committees]  went  as  high  as  126  separate  boards  and  councils”  (21:120). 
The  revision  also  established  Milestone  IV  and  Milestone  V.  Milestone  IV  was  designed 
to  be  a  review  of  the  program  two  to  three  years  after  the  initial  deployment  of  the  system 
to  assure  its  operability  and  supportability,  and  Milestone  V  was  designed  to  determine 
the  operational  effectiveness  five  to  ten  years  after  deployment  (21:120).  Both 
milestones  were  added  to  ease  the  minds  of  critics  who  questioned  the  departments’ 
attention  to  life  cycle  implications  of  new  systems  (21:120). 

2,4.3,5  The  George  H,W,  Bush  Administration  (1989-1993) 

There  was  only  one  revision  to  the  DoD  5000  Series  under  the  Bush 
administration.  However,  those  changes  were  more  noticeable  than  any  other  changes 
made  prior  to  that  date.  The  1991  revision  of  the  DoD  5000  Series  took  place  as  a  result 
of  the  1989  Defense  Management  Repot  (DMR)  authored  by  then  Secretary  of  Defense, 
Dick  Cheney  (21:121).  The  objectives  of  the  1991  revision  were  to  create:  1)  a  uniform 
system  of  acquisition  policy,  2)  provide  rigid  guidelines  for  programs  through  the 
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acquisition  life  cycle-did  not  allow  services  to  supplement  the  DoD  5000  series,  3)  made 
the  DoDI  5000.2  applicable  to  all  acquisition  programs  not  just  major  programs,  and  4) 
mandated  that  all  necessary  information  would  be  transmitted  in  writing  (a  clear 
departure  from  Packard’s  vision  of  less  paperwork).  And  while  previous  versions  failed 
to  exceed  60  pages,  the  1991  version  consisted  of  over  900  pages  (21:122).  The  1991 
version  removed  all  program  flexibility  by  forbidding  waivers  to  the  instructions.  This 
burdened  the  defense  acquisition  process  because  now  paperwork  was  required  for 
everything! 

2,4,3, 6  The  Clinton  Administration  (1993-2001) 

There  were  two  revisions  made  during  the  Clinton  era.  The  first  took  place  early 
in  the  administration  in  1993  and  very  little  changes  were  made.  However,  the  second 
edition  was  released  in  1996.  The  Clinton  administration’s  version  drastically  changed 
the  1991  revisions  to  the  DoD  5000  Series.  It  was  as  if  the  administration  set  out  to  undo 
what  the  Bush  administration  had  done  in  1991.  Ferrara  made  the  following  statement  in 
his  article: 

The  1991  documents  represented  a  dramatic  centralization  of  policy  control  and 
procedural  specificity.  And  the  1996  version  represents  an  equally  dramatic 
reversal  of  these  elements  (21:121)1 

The  1996  revision  was  the  antithesis  of  the  1991  version  as  it  attempted  to  re-instill  the 
Packard  spirit  into  the  regulations.  The  1996  version  reversed  the  decision  to  make  the 
5000.2  applicable  to  all  programs  in  an  attempt  to  give  more  authority  and  flexibility  to 
components  to  run  their  programs  efficiently.  It  also  attempted  to  respond  to  the 
changing  world  environment  brought  on  since  the  end  of  the  Cold  War.  Since  threats  to 
the  United  States  could  come  from  anywhere  at  anytime,  the  acquisition  process  had  to 
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be  flexible  so  that  the  U.S.  eould  respond  rapidly  to  any  threats.  The  1996  edition 

instituted  the  eoneept  of  “Advaneed  Coneept  Teehnology  Demonstrations  (ACTD),” 

Integrated  Produet  Teams  (IPT),  and  it  eaneeled  numerous  reports  required  in  the  1991 

version  (21:123).  The  value  of  these  additions  to  the  1996  version  of  the  DoD  5000 

Series  was  evident  after  the  attaeks  of  September  1 1 ,  200 1 .  The  flexibility  of  the 

aequisition  proeess  was  tested  when  terrorists  hijaeked  three  eommereial  airplanes, 

erashing  one  into  the  World  Trade  Center  in  New  York,  one  in  a  field  in  Pennsylvania, 

and  the  other  into  Pentagon  building  in  Washington  D.C.  Although  many  other  changes 

had  occurred  before  the  attacks,  the  principle  of  flexible  and  responsive  systems  became 

more  apparent  during  that  crucial  period. 

2,4,3.7  The  George  W,  Bush  Administration  (2001-present) 

According  to  an  article  written  for  National  Defense  Online,  president  Bush’s 

Secretary  of  Defense,  Donald  Rumsfeld  demanded  the  “transformation  of  the  Defense 

Department  business  practices,  for  greater  innovation  and  flexibility  in  weapons 

acquisition”  from  the  time  he  stepped  into  office  (20:3).  Dr.  Rumsfeld’s  main  vehicle  for 

codifying  his  “transformation”  was  to  make  changes  to  the  DoD  5000  Series.  In  the 

memo  that  canceled  the  DoDD  5000.1  and  DoDI  5000.2,  dated  2000  and  2002 

respectively,  Rumsfeld’s  Deputy  Secretary  of  Defense  Paul  Wolfowitz  states: 

“I  have  determined  that  the  current  subject  documents  require  revision  to  create 
an  acquisition  policy  environment  that  fosters  efficiency,  flexibility,  creativity  and 
innovation  (42:1).” 

Although  the  DoD  5000  Series  was  canceled  in  October  2002,  the  latest  version  was 
released  on  May  12,  2003.  According  to  the  National  Defense  article,  the  reason  the 
defense  department  sought  to  again  revise  the  DoD  5000  series  in  2003  was  that  previous 
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attempts  at  instilling  flexibility  in  the  regulations,  .  .have  not  gone  far  enough  because 
they  have  not  addressed  adequately  the  need  for  more  innovation  and  efficiency”  (20:1). 
The  author  of  the  article  asserts  that  senior  defense  officials  are  still  frustrated  because, 

“. .  .many  weapons  programs  are  years  behind  schedule,  as  a  result  of  a  cumbersome 
procurement  process,  and  that  acquisition  managers  don’t  work  as  efficiently  as 
commercial  businesses  do,  because  they  are  restricted  by  the  rules”  (20:2).  Erwin 
theorizes  that  despite  all  of  the  previous  revisions  to  the  DoD  5000  Series,  defense 
officials  see  the  instructions  as  requiring  too  much  oversight  and  that  the  oversight 
continues  to  slow  down  the  process.  In  Hawthorne’s  briefing  entitled  “Evolutionary 
Acquisition  Update  and  the  DoD  5000  Revision,  he  summed  up  the  DoD  5000  Series 
pitfalls  prior  to  its  2003  revision.  He  stated  that  the  policies  were  “overly  prescriptive” 
and  did  not  “constitute  an  acquisition  policy  environment  fostering  efficiency,  creativity, 
and  innovation”  (26:14).  However,  he  stated  that  the  objectives  of  the  latest  version  of 
the  DoD  5000  Series  are  to,  “encourage  innovation  and  flexibility;  permit  greater 
judgment  in  the  employment  of  acquisition  principles;  focus  on  outcomes  instead  of 
process;  empower  program  manager’s  to  use  the  system  vice  being  hampered  by 
regulation”  (26:14). 

Over  the  thirty  plus  years  of  the  existence  of  the  DoD  5000  Series,  the  different 
presidential  administrations  attempted  to  implement  procedures  in  the  instructions  that 
would  foster  an  acquisition  environment  of  “efficiency,  flexibility,  creativity,  and 
innovation.”  Although  the  previous  changes  have  only  become  cyclical  points  in  the 
history  of  the  ever  evolving  DoD  5000  Series,  the  outcome  of  the  latest  version  has  yet  to 
be  determined,  but  only  time  will  tell. 
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2.5  Previous  Research 


There  has  not  been  a  lot  of  researeh  that  focuses  on  capturing  the  cost  of  oversight 
of  MDAPs.  However,  there  have  been  studies  that  examined  cost  overruns  or  the  cost  of 
oversight  for  contractors.  This  section  will  discuss  some  of  the  research  that  has  been 
conducted  in  these  areas.  This  research  is  being  conducted  to  fill  the  gap  in  current 
literature.  No  previous  literature  specifically  addresses  the  cost  of  the  vertical  levels  of 
oversight  of  MDAPs  that  accrues  while  an  MDAP  moves  from  one  stage  of  development 
to  the  next.  A  look  at  studies  dealing  more  generally  with  the  topic  of  oversight  of  DoD 
acquisition  follows. 

2.5,1  Contractor  Oversight 

The  cost  of  contractor  oversight  was  the  subject  of  a  1994  study  conducted  by  the 
General  Accounting  Office  (GAO).  The  GAO  report,  printed  in  1997,  entitled 
Acquisition  Reform:  DoD  Faces  Challenges  in  Reducing  Oversight  Costs,  the  GAO 
stated  that  the  results  of  “reinvention  laboratories  which  were  conducted  in  ten  different 
defense  contractor  sites  in  1994  with  an  eye  on  reducing  oversight  costs”  (23:1).  This 
effort  was  one  of  the  major  reforms  to  come  out  of  the  National  Performance  Review  of 
1993.  Each  of  the  test  sites  set  up  functional  evaluation  teams  consisting  of  members 
from  various  different  government  departments,  including  representatives  from  weapon 
systems  program  offices.  Their  objectives  were  to  perform  cost  benefit  analyses  of 
oversight  requirements  and  eliminate  non-value  added  requirements.  It  was  a  large 
undertaking  with  mixed  results.  The  labs’  work  resulted  in  “limited  progress  in 
implementing  changes  to  reduce  contractors’  costs  of  complying  with  government 
regulations  and  oversight  requirements”  (23:4).  They  concluded  that  although  they  still 
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firmly  believed  the  initiatives  were  worthwhile,  great  progress  could  not  be  made  without 
greater  support  from  across  the  DoD.  The  GAO  report  highlights  an  important  part  of  the 
cost  of  oversight  of  acquisition  programs;  however,  it  deals  only  with  the  cost  of 
contractor  oversight,  and  the  costs  of  government  oversight  will  be  evaluated  in  this 
thesis. 

2.5,2  Cost  Overruns 

Over  the  years  different  committees  have  been  formed  to  either  reform  the 
acquisition  process  or  look  for  ways  to  help  reduce  cost  overruns.  Cost  overruns  have 
often  been  a  problem  in  acquisition  and  represent  a  major  reason  why  the  DoD  5000 
Series  is  in  existence  today.  There  have  been  some  studies  done  on  the  effectiveness  of 
the  recommendations  made  by  committees  such  as  the  Packard  Commission  of  1986.  A 
study  conducted  by  Christensen,  Searle,  and  Vickery  reported  in  a  1999  Acquisition 
Review  Quarterly  article  concluded  that  “despite  the  implementation  of  more  than  two 
dozen  regulatory  and  administration  initiatives,  there  has  been  no  substantial 
improvement  in  the  cost  performance  of  defense  programs  for  more  than  30  years 
(8:252).  The  report  used  data  from  the  Defense  Acquisition  Executive  Summary  (DAES) 
database  and  found  that  Packard  Commission  initiatives,  “did  not  reduce  the  average  cost 
overrun  percent  experience  on  269  completed  defense  acquisition  contracts”  (8:251).  Of 
note,  the  study  concluded  that  not  only  were  the  Packard  initiatives  ineffective  in  regards 
to  reducing  cost  overruns,  but  that  overall  cost  overruns  on  the  269  contracts  they 
reviewed  actually  increased  (8:258). 

In  an  effort  to  reduce  cost  the  government  has  implemented  many  initiatives.  In  a 
1994  GAO  report,  it  reported  that  DoD  developed  new  standards  to  reform  the  military 
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specifications  and  standards  in  hopes  of  reducing  acquisition  cost  (22:1).  It  also  stated 
that  the  DoD  military  specification  process  was  complex,  and  often  rigid,  and  blocked  the 
use  of  commercial  products  and  processes  which  prohibited  the  government  from 
reducing  cost  (22:2).  The  reform  being  reported  built  on  previous  studies  in  the  same 
area.  The  report  also  stated  that  most  of  the  recommendations  were  not  new  and  had 
been  suggested  prior  to  this  report.  According  to  the  report,  “this  reform  effort  focuses 
on  changing  the  acquisition  culture  and  contains  several  actions  to  accomplish  this 
change  (22:3).  Like  others  before  it,  the  changes  implemented  have  made  no  significant 
difference  in  cost  overruns  within  the  acquisition  process. 

In  a  1998  article 'm  Acquisition  Review  Quarterly,  Delano  stated  that  the 
“Department  of  Defense  acquisition  programs  and  projects  frequently  experience  cost 
overruns,  performance  deficiencies,  schedule  delays,  or  cancellation”  (12:1).  He  believed 
that  because  the  DoD  had  such  large  programs  that  cost  overruns  were  inevitable  (12:1). 
Delano  set  out  to  find  success  factors  that  DoD  program  managers  could  effectively 
apply  to  enhance  the  success  of  their  acquisition  programs  (12:2).  Delano’s  goal  was  to 
find  success  factors  that  could  be  implemented  quickly  without  policy  changes.  His 
results  did  mirror  that  of  previous  studies,  but  he  was  unable  to  clearly  determine  if  these 
factors  would  in  fact  reduce  the  cost  of  acquisitions. 

2.6  Summary 

This  chapter  presented  the  guidelines  under  which  an  IT  program  as  well  as  a 
normal  program  must  operate.  It  also  defined  the  important  terms  that  will  be  used 
throughout  the  thesis,  and  then  it  went  on  to  discuss  the  evolution  of  the  regulatory  series 
that  establishes  those  guidelines.  The  chapter  also  looked  at  the  DoD  5000  changes  by 
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presidential  administration  and  eoneluded  with  a  look  at  some  previous  researeh  in  the 
area  of  eost  overruns  and  oversight.  There  has  not  been  mueh  studies  done  in  the  area  of 
eost  oversight  of  MDAPs  and  this  thesis  will  be  one  of  the  first  steps  to  determining  that 
eost.  Future  researeh  may  determine  if  that  oversight  is  worth  its  eost. 
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3,0  Methodology 


3.1  Overview 

The  previous  chapters  outlined  changing  oversight  of  Major  Defense  Acquisition 
Programs  under  the  guidelines  provided  in  the  DoD  5000  series  of  regulations.  The 
primary  goal  for  this  research  is  to  estimate  the  cost  of  program  oversight  by  comparing 
the  oversight  of  IT  acquisition  programs  to  the  oversight  of  space  programs  (16:1)  and 
traditional  programs  (35:1)  using  the  Delphi.  In  additional  to  this  goal,  the  following 
additional  questions  are  examined: 

1 .  What  is  the  cost  of  oversight  for  an  IT  program? 

2.  How  does  cost  of  oversight  for  IT  programs  compare  to  other  MDAPs  using 
traditional  and  nontraditional  approaches? 

3.  What  are  the  drivers  for  oversight  costs? 

This  chapter  outlines  the  current  application  of  the  Delphi  Method. 

3.2  Delphi  Method 

This  section  will  provide  some  background  information  on  the  Delphi  Method, 
and  its  history  will  also  be  discussed.  After  the  background  and  history  of  the  Delphi 
Method  is  explored,  a  discussion  of  what  it  is  and  description  of  how  it  works  will  follow. 
Once  a  general  understanding  of  the  Delphi  Method  has  been  given,  it  will  be  obvious 
that  this  methodology  provides  makes  sense  to  evaluate  the  research  topic.  The  final 
section  of  this  chapter  will  go  into  further  detail  on  the  reasons  for  selecting  the  Delphi 
methodology  for  this  research  project.  The  methodology  for  the  execution  of  this 
research  project  will  be  combined  within  of  each  of  the  subject  areas’  description. 
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3.2.1  History  of  the  Delphi  Method 

According  to  Linstone  and  Turoff,  “Delphi  may  be  eharacterized  as  a  method  for 
structuring  a  group  communication  process  so  that  the  proeess  is  effective  in  allowing  a 
group  of  individuals,  as  a  whole,  deal  with  a  complex  problem”  (30:15).  The  word 
“Delphi”  originated  from  Greek  mythology;  it  refers  to  a  Delphi  Oracle  whieh  was 
capable  of  predieting  the  future”  (9:376).  The  Delphi  method  uses  past  experiences  from 
a  panel  of  “experts”  without  group  discussions  to  predict  future  outeomes  (34:17).  This 
methodology  was  developed  in  the  1950s  by  the  The  RAND  Corporation  when 
conducting  a  study  in  support  of  an  Air  Force  exploration  and  the  defense  industry  as  part 
of  a  projeet  ealled  “Projeet  Delphi”  (34:17).  The  U.S.  Air  Foree  wanted  to  determine 
what  would  be  key  nuelear  targets  and  what  would  be  the  likely  number  of  warheads 
employed  against  the  United  States  in  the  event  of  nuclear  attaek  by  the  Soviets.  “Project 
Delphi”  sought  to  reaeh  a  consensus  of  expert  opinion  in  order  to  answer  those  two 
eritieal  questions  from  the  viewpoint  of  a  Soviet  nuclear  strategist.  Because  there  was  a 
lack  of  data  in  this  type  of  researeh  the  Delphi  method  was  ereated. 

3.2.2  What  is  the  Delphi  Method 

The  Delphi  Method  is  best  deseribed  as  a  eommunieation  tool  that  facilitates  a 
communication  process  by  allowing  a  group  of  individuals  to  work  as  a  whole  to  deal 
with  a  problem  (6:701).  The  Delphi  Method  attempts  to  reach  a  consensus  of  opinions 
among  the  members  of  a  group,  which  from  here  on  will  be  referred  to  as  an  expert  panel, 
through  a  series  of  questionnaires.  The  use  of  anonymous  questionnaires  in  the  Delphi 
proeess  eliminates  heated  confrontation  amongst  panel  members  beeause  there  are  no 
faee  to  faee  meetings  or  discussions  throughout  the  proeess  and  replace  them  with  a 
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carefully  planned,  anonymous,  and  orderly  program  of  sequential  individual 
interrogations  usually  eondueted  by  questionnaires”  (40:30).  Anonymous  questionnaires 
allow  eaeh  panel  member  the  freedom  of  expression.  Although  panel  members  are 
provided  with  the  overall  responses,  only  the  administrator  has  knowledge  of  the 
partieipant’s  individual  responses  and  arguments.  The  eharacteristies  of  the  Delphi 
method  is  intended  to  overeome  the  drawbaeks  of  eonventionally  struetured  groups  by 
using  the  anonymous  questionnaire  teehnique  (18:25).  Another  key  element  of  the 
proeess  is  the  panel  of  pre-selected  experts  which  never  has  to  be  physieally  in  the  same 
loeation.  In  the  early  years  of  the  Delphi  proeess,  the  studies  were  eondueted  using  the 
traditional  mailing  system;  however,  in  today’s  virtual  soeiety,  the  studies  ean  be 
eondueted  via  web-based  system,  email  or  a  eombination  of  the  two  methods. 

3.2,3  How  the  Delphi  Method  Works 

The  previous  seetion  of  this  ehapter  offered  a  preliminary  look  at  how  the  Delphi 
Method  works,  but  this  seetion  will  go  into  much  greater  detail  on  the  workings  of  the 
Delphi  Method.  First,  it  is  important  to  answer  some  questions.  The  first  is  why  use  a 
panel  of  experts  that  never  meet  instead  of  just  a  single  expert.  The  reason  is  that  an 
individual  is  operating  alone  whieh  means  they  eould  forget  something  or  fail  to  eonsider 
an  issue.  Clayton  highlighted  this  issue  when  he  discussed  the  faet  that  individuals  don’t 
get  the  benefit  of  hearing  the  ideas  of  others  so  that  they  ean  perhaps  refine  their  ideas 
(9:375).  Clayton  goes  on  to  state  that  by  combining  the  judgment  of  a  large  number  of 
people,  there’s  a  better  ehance  of  arriving  at  the  truth. 

Having  explained  why  a  separated  group  and  not  an  individual,  the  question  then 
beeomes,  if  a  group  is  better  than  an  individual,  wouldn’t  it  be  better  to  put  them  in  a 
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room  together  to  allow  them  to  brainstorm  and  hammer  out  a  consensus?  Though  this 
research  operates  under  Clayton’s  premise  that  the  shared  ideas  of  a  group  of  experts  is 
better  than  a  single  expert,  putting  a  panel  in  a  room  together  could  lead  to  group  think 
(9:375).  This  phenomenon  is  the  result  of  a  few  dominant  personalities  controlling  the 
discussion  and  potentially  strong  arming  a  consensus  despite  the  initial  objections  of 
possibly  better  informed,  yet  more  timid  panel  members.  Anonymity  eliminates  the 
influence  of  the  dominant  individual  and  reduces  both  noise  and  the  pressure  for 
conformity  (18:25). 

Now  that  the  two  preliminary  questions  regarding  the  overall  set  up  of  the  Delphi 
Method  have  been  answered,  the  next  step  is  to  describe  the  workings  of  the  Delphi 
Method.  To  aid  in  this  presentation,  the  key  elements  of  the  workings  of  the  Delphi 
Method  are  explained  best  in  Figure  3.1. 
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Develop  Final  Report 


Figure  3.1:  Oversight  Approval  Levels  (13:2) 

First,  a  problem  is  defined.  For  this  researeh,  the  research  questions  are  the  main 
problems  defined  which  is  to  determine  the  cost  of  oversight  for  IT  programs  as  well  as 
compare  the  costs  of  those  IT  programs  to  the  “box”  and  space  programs.  The  other 
research  focus,  using  the  Delphi  Method  is  to  determine  key  oversight  cost  drivers.  The 
next  step  is  to  develop  a  questionnaire  that  is  specific  enough  to  divulge  the  data 


necessary  to  answer  those  questions.  The  following  step  is  to  select  a  panel  of  experts  to 
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answer  the  questionnaire.  The  questionnaires  are  then  sent  to  the  panel  and  when 
completed  they  are  collected,  analyzed,  and  summarized.  If  consensus  is  not  reached,  the 
summarized  responses  will  then  be  sent  back  out  to  the  panel  to  allow  them  to  rethink  the 
questions  now  that  they  have  the  added  benefit  of  the  input  from  the  other  group 
members.  This  process  of  sending  out  the  questionnaires  and  then  getting  them  back  and 
analyzing  them  continues  in  a  looping  pattern  and  each  loop  is  referred  to  as  a  “round.” 
Each  time  a  new  questionnaire  is  distributed  marks  the  beginning  of  a  new  round.  The 
number  of  rounds  is  determined  by  the  achievement  of  consensus  of  the  expert’s 
opinions.  Early  criticisms  of  the  Delphi  Method  centered  on  the  fact  that  originally,  (due 
to  lack  of  technology)  questionnaires  were  sent  by  traditional  mail  channels  and 
depending  on  the  number  of  rounds  needed  to  achieve  consensus,  the  process  took  from 
several  months  up  to  a  year  or  two  to  complete.  Today’s  technology  enables  the  process 
to  flow  much  more  quickly,  and  for  the  purpose  of  this  particular  research  effort,  all 
communication  during  the  process  will  be  conducted  via  e-mail.  Chou  takes  this  e-mail 
centered  Delphi  methodology  a  step  further  by  conducting  a  web  based  Delphi  Process 
whereby  panel  members  and  the  survey  director  interact  in  a  shared  web  program  (7:233- 
236).  In  summary,  the  Delphi  Method,  as  employed  in  this  research  effort,  will  act  as  a 
communication  facilitator  that  attempts  to  achieve  a  consensus  of  opinions  from  an 
anonymous,  geographically  separated  panel  of  experts  through  a  series  of  questionnaires 
all  conducted  via  e-mail. 

3,2,3, 1.  The  Rounds  of  the  Delphi  Method 

As  previously  mentioned,  each  time  a  questionnaire  is  distributed  to  panel 
members  and  returned  to  the  person  directing  the  research  effort  constitutes  a  round  of 
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the  Delphi  Method.  The  big  question  that  arises  deals  with  how  many  rounds  of  the 
Delphi  are  necessary  to  ensure  the  data  is  stable.  Clayton  states  that  only  four  phases  are 
needed  and  that  the  final  round  is  sent  out  to  “provide  reasons  as  to  why  they  agree  or 
disagree  with  the  final  results”  (9:129).  Chan  et  al.  agreed  in  their  study  by  establishing 
four  rounds  (6:701).  Ludwig  states  that  “Delphi  rounds  continue  until  a  predetermined 
level  of  consensus  is  reached  or  no  new  information  is  gained”  (31:3),  but  a  study  in 
Scotland  by  Dr.  Kerr  limited  the  number  of  rounds  to  3  (28:3).  In  recent  nursing 
research,  Hasson  et  al.  limited  the  number  of  rounds  depending  on  “time  available. . .” 
(25: 101 1).  The  research  did  not  find  a  specific  number  of  rounds  needed.  Most 
researchers  using  the  Delphi  Method  set  the  criteria  of  consensus  and  time  available 
while  some  limited  on  a  firm  number.  Based  on  the  evidence,  the  Delphi  method  as 
employed  in  this  research  effort  to  answer  the  research  questions  will  contain  a  minimum 
of  two  rounds  and  a  maximum  of  four. 

3,2,3,2  Delphi  Method  Questionnaires 

Mitchell  goes  into  great  detail  outlining  the  construction  and  administration  of  the 
Delphi  questionnaires.  He  clearly  outlined  the  length  the  questionnaire  should  be  by 
stating  how  long  it  should  take  each  panel  member  to  complete  the  questionnaire.  On 
this  topic  he  states  that  the  questionnaire  should  take  no  longer  than  30  minutes  to 
complete  (32:345).  The  basis  for  this  assertion  is  his  own  experience  as  he  goes  on  to 
state  that  there  have  been  no  empirical  studies  conducted  on  the  appropriate  length  of 
time  to  complete  a  Delphi  questionnaire.  Mitchell  also  discusses  the  construction  of  the 
questionnaire  for  each  round  of  the  Delphi  Method.  He  states  that  questions  should  be 
clearly  stated  and  should  not  be  identical  from  round  to  round  because  the  repetition 
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could  cause  participant  boredom,  whieh  eould  hamper  results  (32:342).  Clayton  also 
discussed  the  format  of  the  questionnaires  on  a  round  by  round  basis.  He  states  that  the 
round  one  questionnaire  should  be  clearly  worded  but  allow  for  the  most  freedom  in 
responses.  Round  one  responses,  onee  eolleeted,  should  be  turned  into  generie 
statements  summarized  with  measures  of  central  tendeney  and  then  resent  to  panel 
members  to  begin  Round  two.  In  round  two,  the  proeess  of  seeking  eonsensus  begins. 

To  aid  in  the  quest  for  eonsensus  panel  members  that  wish  to  ehange  previous  responses 
must  provide  reasons  for  doing  so.  In  round  three  and  subsequent  rounds,  questionnaires 
should  summarized  responses  with  a  summary  of  reasons  for  ehanging  responses  and  this 
proeess  eontinues  until  eonsensus  is  met  (9:378).  The  questionnaires  in  support  of  this 
researeh  effort  will  be  eonstrueted  aceording  to  the  proeedures  outlined  by  Clayton  and 
Mitehell.  The  number  of  questions  will  be  limited  to  ten  or  less.  The  maximum  amount 
of  time  needed  to  eomplete  eaeh  questionnaire  is  estimated  at  20  minutes.  Eaeh  returning 
questionnaire’s  questions  are  altered  in  each  round  based  on  the  previous  round’s  input. 
This  will  ensure  eaeh  panel  member  has  the  opportunity  to  re-evaluate  eaeh  question. 
3.2,3,3  Delphi  Method  Consensus 

The  rounds  of  questionnaires  must  eventually  eome  to  a  elose.  In  order  to  set  the 
parameters  prior  to  beginning,  onee  eonsensus  is  reaehed,  the  rounds  will  diseontinue. 
Webster’s  New  International  Unabridged  Dietionary  defines  eonsensus  as,  “unanimity  or 
general  agreement  in  matters  of  opinion”  (39:567).  If  that  definition  is  applied  to  the 
Delphi  Method  as  employed  in  this  researeh  effort,  onee  the  panel  reaehes  a  majority 
opinion,  the  proeess  is  eomplete,  but  just  a  majority  may  not  be  far  enough.  Simply 
operating  under  the  theme  of  “majority  rule”  eould  overlook  important,  though  less 
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frequently  oeeurring  opinions.  Therefore,  in  terms  of  the  applieation  of  the  Delphi 
Method  for  this  researeh  effort,  eonsensus  must  be  defined.  The  problem,  as  Williams 
and  Webb  state,  “Consensus  is  poorly  explained  in  studies  whieh  use  the  Delphi 
teehnique...”  (41:182).  Hasson  et  al.  also  state  that:  “A  universally  agreed  proportion 
does  not  exist  for  the  Delphi. . .”  (25: 101 1).  Hasson  et  al.  does  list  various  studies  who 
established  pereentages  for  defining  eonsensus,  but  all  vary  dramatieally  and  result  in 
mostly  a  straight  majority  rules.  A  study  eompleted  by  Sehiebe  et  al.  reeommends 
stability  of  responses  throughout  the  rounds  as  a  better  indieator  of  eonsensus  by 
evaluating  the  ehanges  in  the  questions  to  a  quartile  in  a  distribution  (36:IV:C).  Without 
mueh  empirieal  evidenee  to  support  a  eonerete  definition  of  eonsensus,  this  researeh 
effort  will  take  an  approaeh  similar  to  the  one  reeommended  by  Sehiebe  et  al.  Eaeh 
question  will  be  evaluated  on  the  response  and  as  answers  beeome  stable,  the  question 
will  be  eonsidered  “elosed”  until  all  questions  are  elosed  or  four  rounds  have  been 
eompleted. 

3,2,3,4  Delphi  Method  Expert  Panel 

Another  obstaele  when  performing  the  Delphi  Method  is  deeiding  how  big  the 
expert  panel  should  be.  Spinelli  eondueted  researeh  utilizing  the  Delphi  Method  and  the 
panel  eonsisted  of  “24  key  influential  persons  knowledgeable  as  to  the  faetors  infiueneing 
the  general  environment. . .”  (37:74).  Ludwig  eondueted  researeh  but  had  a  different 
approaeh  to  establishing  a  panel.  Ludwig  stated  that  “The  number  of  respondents  was 
generally  determined  by  the  number  required  to  eonstitute  a  representative  pooling  of 
judgments  and  the  information  summarizing  eapability  of  the  researeh  team”  (31:2).  This 
establishes  the  preeedent  that  as  long  as  all  members  of  the  foeus  researeh  are 
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represented,  the  number  of  members  on  the  panel  is  up  to  the  researeher.  Ludwig  then 
states  “The  majority  of  Delphi  studies  have  used  between  15-20  respondents  and  run  over 
periods  of  several  weeks”  (31:2).  Sinee  it  seems  diffieult  to  find  15-20  volunteers  for  this 
researeh,  further  studies  were  seanned  and  established  more  attainable  preeedents.  Chan 
et  al.  stated  in  their  seleetion  proeess  “The  ten  members  of  the  panel  represent  a  wide 
distribution  of  professional  people. . (6:701).  Another  study  by  Des  Marehais  redueed 
the  panel  size  to  six  (17:504).  But  overall,  William  and  Webb  summarize  the  panel 
seleetion  methodology  by  stating  “First,  there  is  no  agreement  regarding  the  size  of  the 
panel,  nor  any  reeommendations  eoneerning  sampling  teehniques”  (41:182). 

The  panel  assembled  to  answer  the  researeh  questions  posed  in  this  thesis  will  be 
of  the  heterogeneous  type  and  will  embody  the  prineipal  of  breadth  of  members’ 
experienee  while  maintaining  the  similar  target  eareer  field.  The  panel  will  eontain  a 
minimum  of  four  and  a  maximum  of  ten  members. 

Onee  the  size  of  the  panel  has  been  deeided,  a  eriterion  to  determine  exaetly  who 
is  an  expert  has  to  be  established.  Based  on  the  findings  that  were  a  result  of  the  researeh 
eondueted  to  eomplete  this  ehapter,  it  appears  there  is  no  elear  out  definition  of  what 
oonstitutes  an  expert.  While  disoussing  the  topio  of  expert  panel  member  seleetion, 
Mitohell  states,  “No  reported  Delphi  study  has  addressed  this  seleetion  issue”  (32:340). 

In  their  researeh,  Dawson  and  Bruoker  summarized  the  oriteria  for  determining  experts 
used  in  several  Delphi  studies  in  their  field  (11 : 132).  The  oommon  theme  was:  general 
experienee  of  seven  years;  speoifio  experienee  of  five  years;  at  least  one  published  artiole; 
at  least  one  national  oonferenoe  presentation;  and  experienee  should  be  reoent  to  within 
the  last  three  years  (1 1 : 132-134).  For  the  purpose  of  this  researeh,  we’ll  relax  those 
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general  standards  a  bit  by  requiring:  general  experience  of  five  years;  specific  experience 
of  two  years;  recent  experience  within  the  last  five  years;  and  no  qualification  of 
presentations  or  publications. 

Once  the  expert  panel  is  formed,  but  prior  to  the  process  starting,  a  plan  must  be 
instituted  for  panel  attrition.  In  a  study  by  Chan  et  al.  conducted  in  the  field  of  medicine, 
they  achieved  a  response  rate  of  80%  and  went  on  to  state  that  derived  from  various 
studies  that  the  average  response  rate  for  the  medical  field  ranged  from  58%  to  80% 
(6:708).  Mitchell  states  that,  “High  rates  of  attrition  may  mean  that  final  results  are  based 
upon  an  unrepresentative  sub  sample  of  the  original  sample”  (32:341).  To  combat  panel 
attrition  and  the  resulting  degraded  response  rates,  this  research  effort  will  choose  experts 
from  different  but  related  fields  and  have  at  least  one  backup  expert  for  every  expert  so  in 
the  end,  even  with  an  attrition  as  high  as  50%,  all  groups  will  be  represented  and  the  bias 
that  Mitchell  describes  will  be  avoided. 

3.3  Uses  of  the  Delphi  Method 

The  Delphi  Method  has  had  many  uses  in  research.  According  to  the  book  The 
Delphi  Method:  Techniques  and  Applications,  the  Delphi  Method  was  principally  used  as 
a  forecasting  tool  back  as  early  as  the  1960s  and  went  on  to  say  today  the  Delphi  Method 
is  used  for:  normative  forecast;  to  ascertain  values  and  preferences;  quality  of  life 
estimates;  simulated  and  real  decision  making;  and  inventive  planning.  The  book  also 
went  on  to  state  that  the  Delphi  Method  is  used  extensively  where  “judgmental  input 
data”  is  needed  when  other  data  is  unavailable  or  too  costly  (30:615).  Hasson  et  al.  stated 
that  the  Delphi  Method  is  used  frequently  in  health  and  social  sciences  (25:1008). 
Mitchell’s  article  cites  a  table  listing  the  use  of  the  Delphi  method  by  percentage  by  field 
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of  study  from  a  total  of  800  studies.  Delphi  was  most  heavily  used  in  physieal  seienees 
and  engineering  (26%  of  all  studies  eondueted)  and  the  seeond  most  frequent  usage  was 
in  business  and  eeonomies  (23%)  (32:334). 

3.4  Criticisms  of  the  Delphi  Method 

If  employed  properly,  the  Delphi  Method  is  an  exeellent  tool  for  gathering  data  to 
answer  questions  when  that  data  first  appears  to  be  unavailable.  Sinee  this  researeh  effort 
originally  sought  to  analyze  historical  data  and  because  that  data  was  unavailable,  the 
Delphi  Method  appeared  to  be  a  suitable  backup  method.  However  there  are  criticisms  of 
the  Delphi  Method.  The  first  criticism  deals  with  who  actually  decides  what  qualifies  as 
an  “expert”.  Clayton  acknowledges  that  expertise  is  not  exactly  measurable  however,  he 
states  that  the  criteria  is  really  relative  based  on  the  peers  of  the  experts.  For  this  research 
effort,  criteria  for  panelists  will  be  based  criteria  found  in  the  section  on  the  expert  panel 
found  in  this  chapter.  Using  Clayton’s  premise  that  experts  are  deemed  as  such  by  their 
peers,  the  research  will  include  a  preliminary  survey  of  potential  experts.  We’ll  supply 
them  with  our  panel  criteria  and  ask  them  whether  they  agree  with  each  of  the  criteria  or 
not  and  why. 

Williams  and  Webb  introduce  a  second  criticism  of  the  Delphi  Method  which  is 
that  the  researcher’s  analysis  and  summary  of  each  rounds’  responses  could  introduce 
bias  into  the  process  (41:182).  That  point  is  well  taken  and  to  combat  that  threat, 
responses  will  be  analyzed  using  basic  statistical  methods  (mean,  median,  standard 
deviation)  to  the  fullest  extent  possible.  Additionally,  because  this  research  will  conduct 
the  Delphi  Method  as  part  of  a  group  project,  there  will  be  more  than  one  set  of  eyes 
analyzing  the  responses  which  should  also  help  to  keep  the  process  honest. 
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A  final  criticism  of  the  Delphi  Method  regards  the  question  of  reliability; 
speeifieally,  what  evidenee  is  out  there  that  proves  the  Delphi  Method  is  reliable.  In 
other  words,  have  studies  been  eonducted  that  prove  findings  were  consistent  in  different 
Delphi  experiments  using  similarly  eomposed  panels  answering  the  same  questions. 
Williams  and  Webb  found  that,  “there  is  no  evidenee  that  the  Delphi  Method  is  reliable” 
(41:182).  Hasson  et  al.  support  these  findings  stating  that  their  researeh  diseovered, 
“There  is  no  evidenee  of  the  reliability  of  the  Delphi  Method”  (25:1012).  Mitehell  stated 
that  other  studies  have  found  a  high  degree  of  replieability  which  would  contradict  the 
eritieism  that  the  Delphi  Method  is  unreliable  or  at  the  very  least  unproven  (32:351). 

3.5  Strengths  of  the  Delphi  Method 

The  strengths  of  the  Delphi  Method  outweigh  the  weaknesses  previously 
mentioned.  First,  the  Delphi  Method  enables  a  group  of  experts  in  geographieally 
separated  loeations  to  work  together  without  the  eost  or  other  logistieal  problems 
assoeiated  with  bringing  experts  together  at  a  eentral  loeation  (11 : 129).  Anyone  who  has 
tried  to  put  together  a  major  eonferenee  would  greatly  appreeiate  this  strength. 

The  seeond  strength  focuses  on  the  faet  that  the  Delphi  Method  results  in  a 
eonsensus  of  opinion  without  the  bias  or  group  think  that  might  result  from  a  roundtable 
proeess  (41:181).  This  “anonymous  faetor”  ensures  all  panel  members  are  equally 
involved  and  all  panel  members  feel  free  to  answer  honestly.  By  this,  the  researeher  has 
the  opportunity  to  reeeive  uneensored  answers. 

Williams  and  Webb’s  researeh  also  highlights  the  Delphi  eoneept  of  eondueting  a 
series  of  rounds  to  achieve  consensus  (41:181).  The  series  of  rounds  allows  panel 
members  to  review  the  responses  of  their  fellow  panel  members  and  gives  them  the 
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chance  to  reconsider  or  even  alter  their  original  responses  with  the  benefit  of  the  added 
input  of  their  fellow  panel  members.  Condueting  only  one  round  would  destroy  the 
intelleetual  synergy  ereated  by  the  sharing  of  ideas  throughout  the  rounds. 

Finally,  a  eritieism  of  the  traditional  Delphi  proeess  (the  proeess  was  too  long  to 
eomplete)  has  evolved  into  a  strength.  This  long  time  period  was  due  to  the  faet  that  it 
was  used  in  the  1960s  and  1970s  at  a  time  when  there  was  no  means  other  than  through 
postal  ehannels  to  eonduet  Delphi  rounds.  Chou’s  artiele  highlights  the  final  strength  of 
the  Delphi  that  evolved — speed.  Chou  stated  that  traditional  Delphi  proeesses  averaged 
six  to  twelve  months  from  start  to  finish,  but  with  e-mail  and  web-based  Delphi  a  three 
round  study  ean  be  eondueted  in  four  weeks  (7:236). 

3.6  The  Reason  the  Delphi  Method  was  Chosen 

The  originally  theorized  methodology  for  this  researeh  effort  was  to  examine  the 
paper  trail  left  by  an  aetual  MDAP  going  through  a  milestone  deeision  point  i.e.  Meeting 
minutes,  meeting  notes,  sign  in  rosters  to  arrive  at  an  estimated  eost  of  oversight.  Using 
these  doeuments,  the  ranks  and  number  of  people  at  the  meetings  eould  be  aseertained  as 
well  as  the  number  and  duration  of  the  meetings.  This  data  eould  then  be  used  to 
estimate  a  eost  of  meetings  based  on  length  of  meeting  and  the  hourly  wages  of  eaeh 
attendee.  The  estimate  for  meeting  eosts  at  every  level  of  vertieal  oversight  eould  then 
be  tallied  to  arrive  at  a  total  estimate  of  the  eost  of  oversight  for  an  MDAP  at  a  eertain 
key  deeision  point.  The  problem  eneountered  with  this  methodology  was  the  laek  of 
data.  Assuming  that  meeting  minutes,  notes,  and  logs  would  be  readily  available  was  a 
mistake.  In  some  eases  these  items  were  nonexistent.  Therefore,  the  methodology  used 
for  the  purpose  of  this  researeh  had  to  be  able  to  answer  the  researeh  questions  without 
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the  availability  of  historical  data.  An  article  by  V.W.  Mitchell  that  appeared  in 
Technology  Analysis  &  Strategic  Management,  outlines  why  one  would  use  the  Delphi 
Method  with  the  number  on  reason  listed  being  the  unavailability  of  historical  data 
(32:338). 

3.7  Comparative  Analysis  for  Data  Collected 

Once  the  rounds  of  the  Delphi  are  completed,  data  collected  from  this  study  will 
be  compared  to  the  Space  data  collected  by  DeReus  (16:1)  and  the  “Box”  data  collected 
by  Rousseau  (35:1).  The  data  collected  for  all  questions  on  the  questionnaires  will  be 
statistically  analyzed  using  a  software  package  containing  a  graphical  user  interface,  such 
as  JMP  5.0.1  statistical  software.  The  data  will  be  analyzed  to  determine  if  there  are  any 
statistical  differences  in  cost  of  oversight  amongst  the  three  processes. 

3.8  Summary 

This  research  effort  is  aimed  at  answering  the  following  research  questions: 

1 .  According  to  the  panel  of  experts,  what  is  the  cost  of  oversight  for  an  IT 
program  using  the  RIT  process? 

2.  How  does  the  cost  of  oversight  for  an  IT  program  compare  to  the  cost  of 
oversight  for  Box  and  Space  programs? 

3.  What  are  the  top  five  cost  drivers  for  the  oversight  of  IT  acquisition  programs? 
This  chapter  outlined  exactly  how  this  research  effort  will  answer  those  questions. 

In  summary:  the  research  will  consist  of  assembling  a  panel  of  live  to  ten  experts  in  the 
field  of  defense  acquisition;  prepare  questionnaires  aimed  at  collecting  the  cost  of 
oversight  at  one  key  decision  point  and  aimed  at  identifying  oversight  cost  drivers;  then 
employ  the  Delphi  Survey  technique  of  sending  out  the  questionnaires,  collecting, 
analyzing,  summarizing,  and  resending  questionnaires  to  the  panel;  and  continue  with  the 
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Delphi  rounds  until  a  consensus  of  expert  opinion  is  reached.  In  Chapter  4,  the  results  of 
each  round’s  questionnaires  will  be  recorded  and  summarized. 


45 


4,0  Data  Results 


4,1  Overview 

The  goal  of  Chapter  4  is  to  provide  the  results  of  the  data  collected  from  the 
members  of  the  panel  of  experts  used  in  the  Delphi  Method.  In  the  first  section  of  this 
chapter,  only  general  demographics  of  the  panel  members  will  be  provided  to  ensure 
anonymity  of  the  members.  The  next  section  will  provide  the  results  of  the  survey  for 
each  question.  Results  will  be  presented  for  each  of  the  ten  questions  and  will  include 
initial  answers  to  the  questions  and  any  changes  made  during  the  four  separate  rounds  of 
the  Delphi  Method.  The  final  section  will  provide  a  review  of  the  change  in  the  standard 
deviation  for  each  question.  The  chapter  will  conclude  with  the  final  numbers  to  be 
analyzed  and  will  be  used  to  statistically  compare  with  the  results  from  studies  conducted 
by  Rousseau  (35:1)  and  DeReus  (16:1). 

To  establish  the  cost  of  oversight,  an  algorithm  was  created  which  multiplies  and  adds  the 
respondents’  estimates  together  to  create  low,  average,  and  high  estimates  for  the  cost  of 
oversight.  By  multiplying  the  results  of  specific  questions  together,  the  algorithm  works. 
For  example,  to  obtain  a  TDY  cost  estimate,  questions  two,  three,  and  four  are  multiplied; 
to  create  a  personnel  cost  estimate,  questions  five,  six,  and  seven  are  multiplied; 
questions  eight,  nine,  and  ten  are  multiplied  together  to  create  a  meeting  cost  estimate. 
Finally,  to  arrive  at  a  total  program  cost  for  one  milestone  decision  point,  the  estimates 
for  TDY,  personnel,  and  meeting  are  added  together.  The  total  program  cost  for  the  low 
estimate  is  then  represented  by  the  following  algorithm: 

3*((Q21ow*Q31ow*Q41ow)+(Q51ow*Q61ow*Q71ow)+(Q81ow*Q91ow*Q101ow)) 
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The  total  program  cost  for  one  milestone  decision  point  has  to  be  multiplied  by  “3” 
because  there  are  three  milestone  decision  points.  The  same  process  is  used  to  obtain  the 
average  and  high  estimates. 

4.2  Panel  Selection 

The  goal  of  panel  selection  was  to  gather  experts  within  the  IT  community  with 
the  appropriate  acquisitions’  experience.  A  varied  breadth  of  experience  amongst  the 
panel  members  was  also  a  goal  in  order  to  collect  data  from  individuals  with  different 
opinions  because  of  different  experiences.  The  demographics  of  the  selected  panel  of 
experts  are  listed  in  table  4.1  below. 


Table  4.1:  Panel  Selection  Demographics 


Number 

Military/ Civilian 

Position 

Acn  Exo 

(Years) 

IT  Exo 

(Years) 

1 

Military 

Military  Officer 

10 

5 

2 

Civilian 

Retired  Civil  Servant 

30 

30 

3 

Civilian 

Retired  Military  Officer; 

Contractor 

30 

20 

4 

Civilian 

Civil  Servant 

29 

20 

With  the  panel  members  numbered,  the  remainder  of  the  results  and  analysis  will 
refer  only  to  the  number  assigned  for  the  Delphi  Method.  Although  the  table  looks  heavy 
with  civilian  experience,  the  military  viewpoint  is  represented  from  both  the  “worker 
bee”  (Junior  officer)  and  senior  staff  levels  (Retired  0-6).  With  an  average  of  24.75 
years  of  defense  acquisition  experience  and  18.75  years  of  IT  acquisition  experience  this 
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panel  of  experts  has  a  wealth  of  experienee  to  offer.  The  panel  members  have  worked  at 
various  levels  of  the  aequisition  proeess.  Their  experienees  levels  range  from  the 
Program  Offiee  IPT  levels  through  the  OSD  levels  whieh  provides  enough  breadth  of 
experienee  to  provide  the  appropriate  level  of  heterogeneity  referred  to  in  the 
methodology  seetion  of  this  thesis.  The  differenees  of  the  panel  members  will  provide 
the  greatest  probability  of  approaehing  the  true  answer  of  the  unknown  foreeast  we  are 
trying  to  make  and  eompare.  Prior  Internal  Review  Board  permission  was  requested  and 
obtained  for  this  researeh  and  the  letter  of  approval  ean  be  seen  in  the  attaehment  seetion. 
4.3  Question  One 

From  the  Program  Executive  Officer  (PEO)  request  for  a  Defense  Acquisition 

Board  (DAB  to  the  DAB  milestone  approval,  what  are  the  five  major  cost  drivers 

in  the  oversight  process? 

The  goal  of  question  one  was  to  obtain  the  top  five  eost  drivers,  aeeording  to  the 
respondents,  that  drove  the  eost  of  oversight.  There  were  no  stipulations  plaeed  on  the 
answers  to  be  provided  by  the  panel  members;  however,  they  were  limited  to  five 
responses. 

4,3,1  Results  by  Round 

In  round  one,  the  panel  was  asked  to  provide  the  top  five  eost  drivers  within  the 
oversight  proeess  but  they  were  asked  to  put  them  in  no  partieular  order.  In  all,  the  panel 
members  identified  a  total  of  18  eost  drivers,  and  they  are  listed  below: 

1 .  Doeumentation  development 

2.  Doeumentation  review/staffing 

3.  Negotiating  viewpoints  of  the  various  stakeholders. .  .aeq  strategy  re-do 

4.  Changing  oversight  requirements 

5.  After  Milestone  B,  future  milestones  require  less  effort  sinee  mueh  of  the 
groundwork  is  already  eomplete 

6.  Laek  of  oversight  requirement  and  proeess 
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7.  Lack  of  functional  requirements  that  are  elearly  defined  and  understood 

8.  Travel  and  prep  work 

9.  Review  and  approval  of  ORD/Arehiteeture 

10.  Review  and  approval  of  Cost  Estimate 

1 1 .  Review  and  approval  of  Clinger-Cohen  Aet 

12.  Review  and  approval  of  C4ISP 

13.  Review  and  approval  of  TEMP 

14.  Eack  of  defined  and  managed  aequisition  proeesses  in  the  PEO/PMO 

15.  The  need  for  OSD  to  serve  as  MDA  when  there  is  no  defined  or  managed  aeq. 
proeess  in  the  PEO/PMO 

16.  Eaek  of  trust  by  OSD  gate  keepers  in  the  Component  eounterparts  thus 
preventing  them  from  delegating 

17.  The  serial  proeess  of  doeument  approval  by  several  eehelons  of  oversight 

18.  The  laek  of  established  arehiteetures  and  the  resulting  need  for  unique  C4ISP 
efforts 

Eor  round  two,  the  1 8  items  reeeived  from  round  one  were  sent  baek  out  to  the 
panel  members.  They  were  then  asked  to  seleet  the  five  of  the  top  cost  drivers  from  the 
list  and  put  them  in  the  provided  spaee  on  the  survey.  Seven  drivers  were  dropped  from 
the  list  beeause  they  reeeived  no  votes.  The  following  drivers  were  sent  baek  to  the  panel 
members  for  round  two.  They  are  listed  in  no  partieular  order. 

1 .  Documentation  development 

2.  Doeumentation  review/staffing 

3.  Negotiating  viewpoints  of  the  various  stakeholders. .  .aeq.  re-do 

4.  Changing  oversight  requirements 

5.  Eack  of  oversight  requirement  and  proeess 

6.  Eack  of  functional  requirements  that  are  elearly  defined  and  understood 

7.  Travel  and  prep  work 

8.  Eaek  of  defined  and  managed  aequisition  proeesses  in  the  PEO/PMO 

9.  Eack  of  trust  by  OSD  gate  keepers  in  the  Component  eounterparts  thus 
preventing  them  from  delegating 

10.  The  serial  proeess  of  doeument  approval  by  the  several  eehelons  of  oversight 

1 1 .  The  laek  of  established  arehiteetures  and  the  resulting  need  for  unique  C4ISP 
efforts 

As  for  round  three,  there  were  a  total  of  1 1  drivers  sent  baek  to  the  panel  members 
with  the  same  instruetions  given  in  round  2.  This  time,  only  two  items  were  dropped 
beeause  they  reeeived  no  votes.  The  results  from  round  three  are  listed  below: 
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Votes 


1 .  Documentation  Development  1 

2.  Doeumentation  review/staffing  1 

3.  Negotiating  viewpoints  of  the  various  stakeholders. .  .aeq  strategy  re-do  3 

4.  Changing  oversight  requirements  3 

5 .  Lack  of  oversight  requirement  and  proeess  0 

6.  Lack  of  functional  requirements  that  are  clearly  defined  and  understood3 

7.  Travel  and  paperwork  0 

8.  Lack  of  defined  and  managed  aequisition  processes  in  the  PEO/PMO  2 

9.  Lack  of  trust  by  OSD  gatekeepers  in  the  Component  eounterparts  thus 

preventing  them  from  delegating  3 

10.  The  serial  process  of  document  approval  by  the  several  echelons 

of  oversight  2 

1 1 .  The  lack  of  established  arehiteetures  and  the  resulting  need  for  unique 

C4ISP  efforts  2 


Since  the  goal  of  question  one  was  to  identify  only  the  top  five  drivers,  the  drivers 
from  the  round  three  results  that  reeeived  only  one  vote  were  also  eliminated  from  the 
fourth  survey  sent  to  the  panel  members.  With  only  one  round  remaining,  eliminating  the 
drivers  with  the  fewest  votes  was  determined  to  be  the  best  way  to  narrow  the  list  down 
to  one  that  would  meet  the  goal  of  identifying  just  five  drivers.  After  eliminating  those 
drivers  from  the  list,  there  were  seven  remaining  drivers  in  whieh  the  panel  members 
could  make  there  seleetion.  Once  again,  panel  members  were  instructed  to  select  only  the 
five  drivers  they  believed  to  be  the  top  five.  However,  in  the  final  round  they  were  also 
instructed  to  prioritize  their  selections.  By  prioritizing  their  seleetions,  the  data  can  be 
analyzed  to  create  a  ranking  of  the  top  five  eost  drivers.  A  simple  average,  by  adding  the 
values  of  the  votes  and  dividing  by  the  number  of  votes,  would  provide  a  scale  to  rank  the 
cost  drivers.  The  results  from  the  final  round  of  question  one  are  list  below: 
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Votes 

1 .  Negotiating  viewpoints  of  the  various  stakeholders. .  .aeq  strategy  re-do  3 

2.  Changing  oversight  requirements  3 

3.  Lack  of  functional  requirements  that  are  clearly  defined  and  understoodd 

4.  Lack  of  defined  and  managed  acquisition  processes  in  the  PEO/PMO  2 

5.  Lack  of  trust  by  OSD  gatekeepers  in  the  Component  counterparts  thus 

preventing  them  from  delegating  2 

6.  The  serial  process  of  document  approval  by  the  several  echelons 

of  oversight  3 

7.  The  lack  of  established  architectures  and  the  resulting  need  for  unique 

C4ISP  efforts  3 


To  arrive  at  a  final  top  five  list  of  cost  drivers,  the  two  drivers  that  received  only  two 
votes  were  eliminated  from  the  list,  and  the  list  was  prioritized  using  the  simple  average 
technique  described  above.  The  final  prioritized  cost  drivers  for  cost  oversight  in  the  IT 
community  are  listed  below: 

1 .  Lack  of  functional  requirements  that  are  clearly  defined  and  understood 

2.  Changing  oversight  requirements 

3.  The  lack  of  established  architecture  and  the  resulting  need  for  unique  C4ISP 
efforts 

4.  The  serial  process  of  document  approval  by  the  several  echelons  of  oversight 

5.  Negotiating  viewpoints  of  the  various  stakeholders. .  .aeq.  strategy  re-do 

According  to  the  data,  “the  lack  of  clearly  defined  and  understood  functional 
requirements”  is  the  number  one  cost  driver  for  oversight  within  the  IT  community. 

4.4  Question  Two 

From  the  PEO  recommendation,  to  the  DAB  approval  of  the  milestone,  use  your 
professional  judgment  and  estimate  how  many  TDYs  are  taken  by  one  person  to 
get  one  program  through  one  Milestone. 

The  goal  of  question  two  was  to  find  out  how  many  TDYs  are  completed  by  one 
individual  in  one  program  to  get  through  one  milestone.  The  members  were  asked  to 
provide  a  low,  high  and  average,  or  most  likely  occurrence  for  this  portion.  This  will 
allow  us  to  establish  a  triangular  distribution  that  will  be  used  later  for  the  data  analysis 
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portion,  as  well  as  to  allow  us  to  estimate  the  low,  average,  and  high  costs  of  oversight 
for  our  comparison  of  the  three  different  MDAP  processes.  Question  two  sets  up  our 
initial  number  in  our  algorithm  to  calculate  the  first  portion  of  our  cost  of  oversight 
model.  Question  two,  three,  and  four  will  be  multiplied  to  establish  our  travel  estimate 
for  the  cost  of  oversight. 

4,4,1  Question  Two  -  Low  Estimate 

The  low  estimate  for  round  one  ranged  from  two  TDYs  to  twelve  TDYs,  but  that 
range  shrunk  significantly  (seven  to  twelve  TDYs)  by  the  end  of  round  four.  Although 
the  median  stabilized  between  rounds  two  and  three  it  shifted  upwards  to  equal  the  mode 
that  was  set  early  on  at  an  estimate  of  twelve  TDYs.  A  consensus  was  not  reached  for 
this  question,  but  the  data  shows  very  little  difference  between  the  mean,  median,  and 
mode  for  the  total  TDYs.  The  standard  deviation  by  the  end  of  round  four  was  less  than 
three.  The  results  of  the  low  estimates  are  listed  in  Table  4.2,  shown  below. 


Table  4.2:  Question  Two  -  Low  Estimate  Results  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

2  to  12 

12,2,4,12 

7.50 

8.00 

12.00 

5.26 

2 

4  to  12 

12,4,10,12 

9.50 

11.00 

12.00 

3.79 

3 

6  to  12 

12,6,10,12 

10.00 

11.00 

12.00 

2.83 

4 

7  to  12 

12,7,12,12 

10.75 

12.00 

12.00 

2.50 

4,4,2  Question  Two  -  Average  Estimate 


The  ranges  for  the  average,  or  most  likely,  estimate  doubled  that  of  the  low 


estimate  during  rounds  one  through  four.  There  was  a  wide-range  in  the  early  rounds  but 


that  range  was  cut  in  half  by  the  end  of  round  four.  Although  the  median  increased 


throughout  the  process,  the  mode  decreased  from  24  to  20  with  three  of  the  four  panel 
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members  seleeting  20.  Again,  eonsensus  was  not  reaehed  but  the  standard  deviation  was 
minimized.  The  results  are  listed  in  Table  4.3,  shown  below. 


Table  4.3:  Question  Two  -  Average  Estimate  Results  By  Round 

Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

4  to  24 

14.25 

14.50 

24.00 

11.27 

2 

6  to  24 

24,6,15,20 

16.25 

17.50 

NA 

7.76 

3 

8  to  20 

20,8,15,20 

15.75 

17.50 

20.00 

5.68 

4 

9  to  20 

17.25 

20.00 

20.00 

5.50 

4,4,3  Question  Two  -  High  Estimate 


The  high  estimate  followed  suit  with  the  other  two  estimates.  There  was  a 


differenee  of  40  at  the  end  of  round  one  but  that  range  deereased  to  just  18  after  the 


eompletion  of  round  four.  As  the  rounds  eontinued,  the  sizes  of  the  mean,  median,  and 


mode  deereased  as  the  range  deereased  while  the  standard  deviation  was  minimized  to 


the  fullest  by  the  end  of  the  rounds.  Again,  the  panelist  did  not  reaeh  a  eonsensus,  but 


three  panel  members  estimated  the  same  number.  The  results  for  the  question  two-high 


estimate  are  found  below  in  Table  4.4. 


Table  4.4:  Question  Two  -  High  Estimate  Results  By  Round 


Frequency 


Round 

Range 

(Member  1, 2,3,4) 

Mean 

Median 

Mode 

Std  Dev 

1 

8  to  48 

48,8,10,48 

28.50 

29.00 

48.00 

22.53 

2 

10  to  48 

48,10,20,30 

27.00 

25.00 

NA 

16.21 

3 

12  to  30 

23.00 

25.00 

30.00 

8.72 

4 

12  to  30 

25.50 

30.00 

30.00 

9.00 

4,5  Question  Three 


Estimate  how  many  people  normally  go  TDY  throughout  the  Milestone  Decision 
process. 


Question  three  established  another  portion  of  the  travel  estimate  in  our  eost  of 


oversight  algorithm.  The  goal  of  question  three  is  to  find  the  number  of  personnel  that 


aetually  go  TDY  during  the  milestone  deeision  proeess.  The  respondents  were  given  the 
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same  instructions  as  was  given  with  question  two  and  the  answers  will  be  presented  in  the 


similar  manner. 

4,5,1  Question  Three  -  Low  Estimate 

Question  three’s  low  estimate  outcomes  were  not  much  difference  from  that  of 


question  two.  Once  again,  the  range  from  round  one  to  round  four  decreased;  however, 
take  notice  that  member  one  and  member  four  were  the  outliers.  Member  one  began  with 
a  low  estimate  and  kept  it  low  throughout  the  Delphi  process.  Member  four  started  out 
high  and  although  the  member  dropped  from  48  to  20,  there  is  still  a  significant 
difference  from  the  final  estimate  of  member  one.  This  is  just  one  example  of  how  the 
Delphi  process  is  not  completely  successful  at  shortening  the  range  of  answers  from  the 
various  experts.  The  various  experiences  of  the  panel  members  could  be  reasoning  for 
the  differences  also.  In  the  end,  both  the  median  and  mode  equaled  five,  and  the  mean 
was  less  than  nine.  The  standard  deviation  is  a  bit  high  compared  to  the  ending  range  but 
that  is  because  of  the  two  outliers  mentioned  earlier.  Once  again,  no  consensus  was 
reached.  The  results  of  question  three-low  estimate  are  listed  below  in  Table  4.5: 


Table  4.5:  Question  Three  -  Low  Estimate  Results  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

3  to  48 

3,3,5,48 

14.75 

4.00 

3.00 

22.19 

2 

3  to  40 

3,3,5,40 

12.75 

4.00 

3.00 

18.19 

3 

3  to  40 

3,4,5,40 

13.00 

4.50 

3.00 

18.02 

4 

4  to  20 

4,5,5,20 

8.50 

5.00 

5.00 

7.68 

4,5,2  Question  Three  -  Average  Estimate 

The  average  estimate  also  had  a  wide  range  of  values  throughout  the  process. 


Unlike  the  low-estimates,  the  average  estimates  only  had  one  panel  member  with  a  vast 


difference  of  opinion.  While  three  of  the  panel  members  estimated  10  or  below,  panel 
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member  four  started  out  extremely  higher  than  the  other  members  of  the  panel.  In  the 
end,  member  four  was  still  at  least  threes  times  higher  than  that  of  the  other  panel 
members.  Beeause  of  the  extreme  differenee,  the  standard  deviation  is  somewhat  high  in 
eomparison  to  the  previous  estimate.  In  this  ease,  the  safest  estimate  would  have  to  be 
the  mean. 


Table  4.6:  Question  Three  -  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

6  to  70 

6,6,10,70 

23.00 

8.00 

6.00 

31.39 

2 

6  to  60 

10,6,10,60 

21.50 

10.00 

10.00 

25.74 

3 

6  to  60 

10,6,10,60 

21.50 

10.00 

10.00 

25.74 

4 

7  to  30 

10,7,10,30 

14.25 

10.00 

10.00 

10.59 

4,5,3  Question  Three  -  High  Estimate 


The  high  estimate  results  listed  in  table  4.7  show  almost  the  same  pattern  that 


oeeurred  with  the  low  and  average  results.  There  was  a  wide  range  of  estimates  from  8  to 


100  in  the  first  round  but  in  the  final  round,  it  ranged  from  12  to  40.  The  mean,  median. 


and  mode  hovered  around  the  low  twenties  and  a  eonsensus  was  not  reaehed.  The  results 


ended  with  a  fairly  high  standard  deviation  at  1 1 .94. 


Table  4.7:  Question  Three  -  High  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1, 2,3,4) 

Mean 

Median 

Mode 

Std  Dev 

1 

8  to  100 

9,8,20,100 

34.25 

14.50 

NA 

44.17 

2 

8  to  80 

20,8,20,80 

32.00 

20.00 

20.00 

32.50 

3 

12  to  80 

20,12,20,80 

33.00 

20.00 

20.00 

31.56 

4 

12  to  40 

20,12,20,40 

23.00 

20.00 

20.00 

11.94 

4,6  Question  Four 


What  is  your  estimate  of  the  cost  for  each  person  on  each  TDY? 


Question  four  provides  the  final  number  for  the  travel  portion  of  the  eost  of 
oversight  formula.  By  multiplying  the  estimates  from  questions  two,  three,  and  four,  an 
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estimate  for  the  eost  of  travel  in  the  oversight  proeess  ean  be  obtained.  Question  four 
will  provide  an  aetual  dollar  figure  estimate  for  the  eost  of  one  TDY  for  one  person  on  a 
team.  Results  are  presented  in  the  same  format  as  previous  questions. 

4.6,1  Question  Four  -  Low  Estimate 

Question  four’s  low  estimate  began  with  a  somewhat  small  range  eompared  to 
that  of  previous  questions,  and  elosed  out  with  a  margin  of  only  $200.  Three  of  the  four 
panel  members  estimated  the  TDY  eost  per  person  at  $1,000  whieh  made  the  median  and 
mode  equal  to  one  another.  The  standard  deviation  was  $100.  The  panel  was  only  one 
member  shy  of  reaehing  a  eonsensus.  All  of  the  results  ean  be  seen  in  Table  4.8  below: 

Table  4.8:  Question  Four  -  Low  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

$500  to  $2000 

$2000, $1000, $500.$1000 

$1,125.00 

$1,000.00 

$1,000.00 

$629.15 

2 

$1000  to  $2000 

$2000, $1000, $1000,$1000 

$1,250.00 

$1,000.00 

$1,000.00 

$500.00 

3 

$1000  to  $1200 

$1000, $1200, $1000,$1000 

$1,050.00 

$1,000.00 

$1,000.00 

$100.00 

4 

$1000  to  $1200 

$1000, $1200, $1000,$1000 

$1,050.00 

$1,000.00 

$1,000.00 

$100.00 

4,6,2  Question  Four  -  Average  Estimate 

The  range  for  the  average  estimate  is  slightly  higher  than  that  of  the  low  estimates 
in  the  beginning  round,  but  the  range  eloses  substantially  by  the  fourth  round.  And  again, 
the  estimates  are  just  one  panel  member’s  estimate  shy  of  reaehing  a  eonsensus. 

However,  the  standard  deviation  deereased  from  $1,314  to  only  $250  whieh  is  an 
aeeeptable  margin.  The  results  are  shown  in  Table  4.9  below. 


Table  4.9:  Question  Four  -  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1, 2,3,4) 

Mean 

Median 

Mode 

Std  Dev 

1 

$1000  to  $4000 

$4000,$2000,$1000,$1500 

$2,125.00 

$1,750.00 

NA 

$1,314.98 

2 

$2000  to  $4000 

$4000,$2000,$2000,$2000 

$2,500.00 

$2,000.00 

$2,000.00 

$1,000.00 

3 

$1500  to  $2000 

$2000, $1 500,$2000,$2000 

$1,875.00 

$2,000.00 

$2,000.00 

$250.00 

4 

$1500  to  $2000 

$2000, $1 500,$2000,$2000 

$1,875.00 

$2,000.00 

$2,000.00 

$250.00 
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4,6,3  Question  Four  -  High  Estimate 


The  high  estimate  for  question  four  had  a  relatively  large  range,  seen  in  Table 


4.10,  but  the  range  constantly  reduced  throughout  all  four  rounds.  The  panel  members 


had  not  reached  a  consensus.  There  were  three  different  estimates  at  the  end  of  round 


four.  Results  from  the  high  estimates  can  be  seen  in  Table  4.10. 


Table  4.10:  Question  Four  -  High  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

$1500  to  $6000 

$6000, $3000, $1 500, $2000 

$3,125.00 

$2,500.00 

NA 

$2,015.56 

2 

$3000  to  $6000 

$6000, $3000,$3000, $3000 

$3,750.00 

$3,000.00 

$3,000.00 

$1,500.00 

3 

$3000  to  $5000 

$5000, $3000,$3000, $4000 

$3,750.00 

$3,500.00 

$3,000.00 

$957.43 

4 

$3000  to  $4000 

$4000, $3000,$3500, $4000 

$3,625.00 

$3,750.00 

$4,000.00 

$478.71 

4,6,4  Travel  Cost  Estimate 


With  all  the  necessary  data  collected,  it  is  now  possible  to  develop  an  overall 
estimate  for  travel  costs  using  the  previously  mentioned  algorithm.  To  review,  the  travel 
cost  estimate  will  be  developed  by  multiplying  the  estimates  from  question  two;  the 
number  of  TDYs  taken  by  one  person,  by  the  estimates  from  question  three;  the  total 
number  of  persons  who  go  TDY,  by  the  estimates  from  question  four;  the  cost  per  person 
for  each  TDY.  The  results  can  be  seen  below  in  Table  4.11. 


Table  4.1 1:  Estimates  of  Travel  Cost  for  One  Milestone 


Questions  2-4 


MEMBER  Travel-LOW  Travel-AVG  Travel-HIGH 


1 

$48,000.00 

$400,000.00 

$2,400,000.00 

2 

$42,000.00 

$94,500.00 

$432,000.00 

3 

$60,000.00 

$400,000.00 

$2,100,000.00 

4 

$240,000.00 

$1,200,000.00 

$4,800,000.00 

MEAN 

$97,500.00 

$523,625.00 

$2,433,000.00 

STD  DEV 

$95,294.28 

$473,355.99 

$1,799,876.66 

The  estimates  were  calculated  by  member  for  low,  average  and  high  and  then  a 
mean  and  standard  deviation  was  calculated  for  all  five  members  for  the  low,  average. 
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and  high  estimates.  Based  on  the  mean,  the  end  result  of  four  rounds  of  the  Delphi  was  a 
travel  estimate  that  ranges  from  $97. 5K  to  $2.4M  for  one  milestone. 

4.7  Question  Five 

Estimate  how  many  hours  are  spent  on  support  for  the  DAB  approval  process  per 
person,  not  including  TDY  travel  time,  but  actual  job  performance  while  TDY  or 
at  home  base.  (Slide  prep,  meeting  prep,  etc) 

In  question  five,  a  new  segment  of  the  eost  of  oversight  algorithm  is  started. 
Question  five  is  the  beginning  of  the  personnel  portion  of  the  estimate.  With  question 
five,  the  goal  is  to  find  the  number  of  hours  personnel  put  in  direetly  towards  the  DAB 
proeess. 

4,7.1  Question  Five  -  Low  Estimate 

The  results,  shown  in  Table  4.12,  started  off  with  a  range  of  12  which  appears  to 
be  a  moderate  range.  The  range  stabilized  by  round  three,  but  the  panelists  were  still 
unable  to  reach  a  consensus  by  the  end  or  round  four.  With  a  standard  deviation  of  two, 
the  group  managed  to  get  three  of  the  four  members  to  agree  on  the  same  number  of 
hours  spent  supporting  the  DAB.  The  median  and  mode  were  equal,  and  they  were  only 
one  off  from  the  mean.  The  complete  results  can  be  seen  in  Table  4.12: 

Table  4.12:  Question  Five  -  Low  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

4  to  16 

4,8,15,16 

10.75 

11.50 

NA 

5.74 

2 

8  to  16 

10,8,15,16 

12.25 

12.5 

NA 

3.86 

3 

10  to  16 

12 

11 

10 

2.83 

4 

12  to  16 

13 

12 

12 

2.00 

4,7,2  Question  Five  -  Average  Estimate 

The  average  estimates  for  question  five  is  completely  inconsistent  throughout  all 
of  the  rounds.  The  range  of  estimates  from  rounds  one  and  four  change  by  only  six 
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hours.  There  was  no  mode  during  either  of  the  rounds  although  the  mean  and  median 
were  equal  by  round  four.  The  panel  members  were  adamant  about  their  initial  estimates 
and  did  not  make  many  changes  to  their  first  instinct.  Only  panel  member  one  changed 
their  initial  estimate  more  than  once  throughout  the  rounds.  Table  4.13  shows  complete 
results  below. 


Table  4.13:  Question  Five  -  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

8  to  24 

8,16,20,24 

17.00 

18 

NA 

6.83 

2 

15  to  24 

15,16,20,24 

18.75 

18 

NA 

4.11 

3 

17  to  24 

17,20,18,24 

19.75 

19 

NA 

3.10 

4 

14  to  24 

18,14,20,24 

19 

19 

NA 

4.16 

4.7,3  Question  Five  -  High  Estimate 

As  with  the  average  estimates,  the  high  estimates  are  very  unstable.  The  range 
decreased  by  fourteen  hours  over  the  course  of  the  Delphi  process,  but  the  panel  members 
were  still  unable  to  come  to  a  consensus.  For  the  high  estimate,  there  was  a  modal  value 
of  thirty  hours.  The  mean,  median  and  modal  values  were  all  within  four  hours  of  one 
another,  and  the  standard  deviation  closed  out  at  4.90.  Further  results  are  shown  in  Table 
4.14. 


Table  4.14:  Question  Five  -  High  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1, 2,3,4) 

Mean 

Median 

Mode 

Std  Dev 

1 

16  to  40 

16,24,30,40 

27.50 

27.00 

NA 

10.12 

2 

24  to  40 

25,24,30,40 

29.75 

27.5 

NA 

7.32 

3 

25  to  40 

31.25 

30 

30 

6.29 

4 

30  to  40 

34 

33 

30 

4.90 
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4.8  Question  Six 


Estimate  how  many  people  are  normally  involved  with  the  preparation  process. 

Question  six  places  an  actual  number  of  personnel  into  the  second  portion  of  the 
algorithm  for  cost  of  oversight.  The  number  of  personnel  involved  in  the  preparation 
process  included  those  creating  slides,  preparing  briefings,  and  supporting  the  DAB.  The 
results  are  given  in  similar  format  as  previous  data  collected. 

4,8,1  Question  Six  -  Low  Estimate 


The  low  estimate  had  a  moderate  range  at  the  end  of  round  one.  However,  the 
range  began  to  decrease  in  round  two  and  by  round  three  the  panelists  were  very  close  to 
a  reaching  consensus.  By  round  four,  the  panel  members  had  reached  its  first  consensus 
during  the  process.  Although  it  is  not  shown  below  in  the  table,  the  final  estimate  for  all 
of  the  panel  members  was  6  hours.  Below  in  Table  4.15,  the  results  are  shown. 


Table  4.15:  Question  Six  -  Low  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

3  to  15 

3,4,15,4 

6.50 

4.00 

4.00 

5.69 

2 

3  to  10 

5,3,10,6 

6.35 

5.50 

NA 

2.63 

3 

5  to  6 

6,5,6,6 

5.75 

6.00 

6.00 

0.50 

4 

Reached  Consensus  | 

4,8,2  Question  Six  -  Average  Estimate 


Again  the  panel  reached  an  overall  consensus,  but  this  time  they  did  so  at  the  end 


of  round  three.  They  determined  that  for  the  average  estimate,  10  people  were  normally 


involved  with  the  preparation  process.  All  of  the  results  can  be  seen  below  in  Table  4.16. 


60 


Table  4.16:  Question  Six  -  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

6  to  20 

6,8,20,6 

10.00 

7.00 

6.00 

6.73 

2 

8  to  15 

10,8,15,10 

10.75 

10.00 

10.00 

2.99 

3 

Reached  Consensus  | 

4,8,3  Question  Six-High  Estimate 

The  high  estimate,  seen  below  in  Table  4.17,  also  ended  up  with  a  eonsensus  by 


the  end  of  round  three.  The  panel  deeided  that  the  high  estimate  for  the  number  of  people 


involved  in  preparation  was  15. 


Table  4.17:  Question  Six  -  High  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1, 2,3,4) 

Mean 

Median 

Mode 

Std  Dev 

1 

9  to  25 

9,12,25,12 

14.50 

12.00 

12.00 

7.14 

2 

12  to  20 

15,12,20,14 

15.25 

14.50 

NA 

3.40 

3 

Reached  Consensus 

4,9  Question  Seven 

Estimate  the  cost  per  hour  for  each  person  involved  in  the  process. 

Question  Seven  provides  the  last  portion  of  the  personnel  eost  estimate  for  the 
total  eost  of  oversight.  With  the  estimates  provided  in  question  seven,  multiplied  by  the 
estimates  given  in  questions  five  and  six,  the  estimated  foreeast  for  the  eost  of  personnel 
in  the  oversight  proeess  ean  be  determined. 


4,9,1  Question  Seven  -  Low  Estimate 


In  Table  4.18,  the  low  estimates  are  provided.  The  estimates  ehanged  in  the  first 


two  rounds  and  remained  unchanged  for  the  last  two  rounds.  The  standard  deviation  was 


reduced  and  leveled  out  after  round  three.  The  mean,  median,  and  mode  remained 


constant  after  round  two.  In  the  final  round  the  mean,  median  and  mode  were  all 


different,  but  were  within  a  total  of  less  than  thirteen  dollars. 
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Table  4.18:  Question  Seven  -  Low  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

$50  to  $100 

1  $60, $100, $50, $80  1 

$  72.50 

$  70.00 

NA 

$  22.17 

2 

$70  to  $100 

$  82.50 

$  80.00 

$  80.00 

$  12.58 

3 

$70  to  $100 

1  $70,$100,$100,$80  1 

$  87.50 

$  90.00 

$  100.00 

$  15.00 

4 

$70  to  $100 

$  87.50 

$  90.00 

$  100.00 

$  15.00 

4,9,2  Question  Seven  -  Average  Estimate 


Although  the  ranges  for  question  seven’s  average  estimates  begin  a  bit  larger  than 


that  of  the  low  estimates,  the  margin  at  the  end  was  much  smaller.  By  the  end  of  round 


two,  the  panel  members  had  almost  reached  a  consensus,  but  the  estimates  of  member 


four  were  very  consistent  throughout  the  process  up  to  round  three.  Member  four  only 


made  changes  to  the  original  estimate  in  round  four.  In  the  end,  the  mean  was  less  than 


four  dollars  shy  of  the  median  and  mode,  and  the  standard  deviation  minimized  to  only  5. 


All  of  the  results  can  be  seen  in  Table  4.19. 


Table  4.19:  Question  Seven  -  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3,4) 

Mean 

Median 

Mode 

Std  Dev 

1 

$70  to  $150 

$  123.75 

$  137.50 

$  150.00 

$ 

37.72 

2 

$120  to  $150 

1  $150,$150,$120,$125  1 

$  135.25 

$  137.50 

$  150.00 

$ 

16.01 

3 

$125  to  $150 

$  143.75 

$  150.00 

$  150.00 

$ 

12.50 

4 

$140  to$150 

1  $150,$150,$150,$140  1 

$  147.50 

$  150.00 

$  150.00 

$ 

5.00 

4,9,3  Question  Seven  -  High  Estimate 


The  high  estimate  for  question  seven  started  out  with  a  moderate  range.  There 


was  not  a  mode  in  rounds  one  in  two,  but  a  mode  was  established  and  stabilized  in  round 


three.  Over  the  four  rounds  both  the  mean  and  median  grew,  but  the  standard  deviation 


fell  throughout  the  process  and  finally  minimized  in  round  four  at  47.26.  One  thing  to 


notice  about  the  high  estimate  was  that  most  members  made  very  few  changes  to  their 
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initial  estimates.  Needless  to  say,  a  consensus  for  the  high  estimate  was  not  reached. 
More  details  on  the  results  can  be  seen  in  Table  4.20. 


Table  4.20:  Question  Seven  -  High  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

$100  to  $300 

1  $300, $200, $100, $220  | 

$  205.00 

$  210.00 

NA 

$ 

82.26 

2 

$150  to  $300 

$  217.50 

$  210.00 

NA 

$ 

62.38 

3 

$200  to  $300 

1  $300,$200,$200,$220  | 

$  230.00 

$  210.00 

$  200.00 

$ 

47.61 

4 

$200  to  $300 

$  235.00 

$  220.00 

$  200.00 

$ 

47.26 

4,9.4  Personnel  Cost  Estimate 


With  all  the  necessary  data  collected,  it  is  now  possible  to  develop  an  overall 
estimate  for  personnel  costs  using  the  previously  mentioned  algorithm.  The  personnel 
cost  estimate  will  be  developed  by  multiplying  the  estimates  from  question  five;  the 
number  of  hours  one  person  spends  in  support  of  a  milestone  review,  by  the  estimates 
from  question  six;  the  total  number  of  persons  who  support  a  milestone  review,  by  the 
estimates  from  question  seven;  the  cost  for  one  person  involved  in  the  milestone  review. 
The  results  can  be  seen  below  in  Table  4.21. 


Table  4.21:  Estimates  of  Personnel  Cost  Eor  One  Milestone 


Questions  5-7 


MEMBER  Person-LOW  Person-AVG  Person-HIGH 


1 

$5,040.00 

$27,000.00 

$135,000.00 

2 

$7,200.00 

$25,200.00 

$129,600.00 

3 

$7,200.00 

$30,000.00 

$90,000.00 

4 

$7,680.00 

$33,600.00 

$144,000.00 

MEAN 

$6,780.00 

$28,950.00 

$124,650.00 

STD  DEV 

$1,181.86 

$3,678.31 

$23,851.42 

The  estimates  were  calculated  by  member  for  low,  average  and  high  and  then  a 


mean  and  standard  deviation  was  calculated  for  all  five  members  for  the  low,  average. 


and  high  estimates.  Based  on  the  mean,  the  end  result  of  four  rounds  of  the  Delphi  was  a 
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personnel  cost  estimate  that  ranges  from  nearly  $6.8K  to  almost  $125K  for  one 
milestone. 


4.10  Question  Eight 

Estimate  how  many  meetings  are  normally  held  from  the  PEO  preparation, 
through  DAB  approval.  (This  includes  meetings  while  TDY or  TDY prep 
meetings). 

Question  eight  provides  insight  on  how  meetings  are  included  into  the  oversight 
process.  By  multiplying  questions  eight,  nine,  and  ten,  we  will  get  an  idea  of  what  part 
meetings  play  in  the  cost  of  oversight.  Question  eight  deals  specifically  with  the  number 
of  meetings  that  are  held  during  one  milestone  in  a  program.  The  results  are  listed  in  the 
following  paragraphs. 

4,10,1  Question  Eight  -  Low  Estimate 

The  low  estimate  for  meetings  held  for  PEO  preparation  began  with  a  very  wide 
range.  The  gap  closed  some  throughout  the  Delphi  process,  but  there  was  not  a 
consensus  in  the  end.  Although  a  consensus  was  not  reached,  a  mode  of  15  was 
established  by  the  end  of  round  four  and  the  standard  deviation  continued  to  decrease.  At 
the  end,  the  standard  deviation  minimized  at  4.24.  The  results  of  the  low  estimates  can 


be  seen  in  its  entirety  in  Table  4.22. 


Table  4.22:  Question  Eight  -  Eow  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

3  to  48 

48,3,10,9 

17.50 

9.50 

NA 

20.57 

2 

6  to  20 

20,6,10,18 

13.50 

14.00 

NA 

6.61 

3 

7  to  20 

20,7,10,18 

13.75 

14.00 

NA 

6.24 

4 

8  to  18 

15,8,15,18 

14.00 

15.00 

15.00 

4.24 
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4,10,2  Question  Eight  -  Average  Estimate 

Again,  the  average  estimates  compared  to  that  of  the  low  estimates  at  the 
beginning  of  the  process.  However,  the  average  estimates  never  showed  any  signs  of 
stabilization.  A  mode  was  never  established  and  the  standard  deviation  minimized  at  a 
high  12.18.  The  range  (12  to  40)  was  still  considerably  high  after  the  four  rounds.  Table 
4.23  illustrates  the  lack  of  stability  in  the  estimates. 

Table  4.23:  Question  Eight  -  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

6  to  100 

100,6,20,12 

34.50 

16.00 

NA 

44.04 

2 

1 0  to  75 

75,10,20,34 

34.75 

27.00 

NA 

28.58 

3 

12  to  50 

50,12,20,34 

29.00 

27.00 

NA 

16.69 

4 

12  to  40 

40,12,25,34 

27.75 

29.50 

NA 

12.18 

4,10,3  Question  Eight  -  High  Estimate 


The  high  estimates  for  question  eight  are  seen  in  Table  4.24.  As  the  estimates  for 


the  low  and  average  estimates,  the  high  estimates  are  just  as  unstable.  The  estimates 


range  from  nine  to  two  hundred  in  round  one,  and  a  mode  is  never  established  for  the 


estimates.  While  the  standard  deviation  stood  at  28,  the  mean  and  median  were  50  and 


52  respectively.  No  consensus  was  reached  in  any  of  the  estimates  for  question  eight,  but 


the  results  from  the  high  estimates  can  be  found  in  Table  4.24. 


Table  4.24:  Question  Eight  -  High  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3,4) 

Mean 

Median 

Mode 

Std  Dev 

1 

9  to  200 

200,9,30,18 

64.25 

24.00 

NA 

90.91 

2 

15  to  150 

150,15,25,64 

63.50 

44.50 

NA 

61.42 

3 

15  to  100 

100,15,30,64 

52.25 

47.00 

NA 

37.86 

4 

16  to  80 

80,16,40,64 

50.00 

52.00 

NA 

28.00 
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4.11  Question  Nine 

What  do  you  estimate  as  the  length,  in  hours,  for  each  meeting? 

Question  nine  provides  a  length  for  each  meeting,  which  will  be  multiplied  by  the 
number  of  meetings,  provided  in  question  eight  and  the  cost  per  hour  for  each  person 
attending,  which  will  be  provided  in  question  ten.  Question  nine  was  an  estimate  that 
came  in  with  relatively  low  standard  deviations  for  each  estimate,  but  overall,  was  not 
significantly  volatile  from  one  round  to  the  next.  Results  are  provided  in  the  following 
three  sub-paragraphs. 

4,11.1  Question  Nine  -  Low  Estimate 

The  range  for  the  low  estimate  in  question  nine  was  quite  small  (3)  and  was 
reduced  even  more  by  the  end  of  the  process.  Only  panel  members  changed  their  initial 
estimates  throughout  the  rounds,  but  member  2  was  the  only  one  that  made  changes  after 
round  two.  The  panel  members  were  short  one  estimate  from  a  consensus  in  the  end  and 
the  margin  of  separation  of  the  estimates  was  only  .5.  The  median  and  mode  equaled 
each  other  and  the  mean  differed  by  .  12.  The  standard  deviation  was  also  very  small.  All 
of  the  results  can  be  seen  in  Table  4.25. 


Table  4.25:  Question  Nine  -  Low  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

1  to  4 

4, 1,2, 2 

2.25 

2.00 

2.00 

1.26 

2 

1  to  2 

2,1,2,2 

1.75 

2.00 

2.00 

0.50 

3 

1 .25  to  2 

2,1.25,2,2 

1.81 

2.00 

2.00 

0.38 

4 

1.5  to  2 

1.88 

2.00 

2.00 

0.25 

4.11.2  Question  Nine  -  Average  Estimate 


Question  nine’s  average  estimate  should  a  similar  pattern  in  the  range  since  it  too 


was  quite  small.  However,  the  two  estimates  were  different  because  there  was  little 
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changes  made  from  the  initial  estimates  throughout  the  proeess  and  there  were  no  same 
estimates.  Therefore,  no  mode  was  established.  There  is  little  differenee  in  the  mean  and 
median,  and  the  standard  deviation  is  also  low.  The  statistics  are  favorable  only  because 
the  range  was  small  to  begin  with  not  beeause  the  members  had  a  ehange  of  opinion.  All 
of  the  results  can  be  seen  below  in  Table  4.26. 


Table  4.26:  Question  Nine  -  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

2  to  8 

8, 2, 4, 3 

4.25 

3.50 

NA 

2.63 

2 

2  to  6 

6, 2, 4, 3 

3.75 

3.50 

NA 

1.71 

3 

2.5  to  5 

5, 2.5, 4, 3 

3.63 

3.50 

NA 

1.11 

4 

2.5  to  5 

5, 2.5, 4, 3 

3.63 

3.50 

NA 

1.11 

4,11,3  Question  Nine  -  High  Estimate 


The  high  estimate  for  question  nine,  loeated  in  Table  4.27  below,  had  a  much 


wider  range  than  that  of  the  low  and  average  estimates  previously  shown.  Although  the 


mean,  median,  and  mode  were  equal  and  the  standard  deviation  was  at  a  low  3.27,  the 


range  was  a  bit  wider  than  that  of  the  low  and  average  estimates.  A  consensus  was  not 


met  for  the  high  estimate  but  it  would  be  safe  to  estimate  eight  hours  as  a  high  estimate 


for  the  length  of  a  meeting. 


Table  4.27:  Question  Nine  -  High  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3,4) 

Mean 

Median 

Mode 

Std  Dev 

1 

3  to  20 

20,3,8,8 

9.75 

8.00 

8.00 

7.23 

2 

3  to  15 

15,3,8,8 

8.50 

8.00 

8.00 

4.93 

3 

3.5  to  15 

15,3.5,8,8 

8.63 

8.00 

8.00 

4.75 

4 

4  to  12 

12,4,8,8 

8.00 

8.00 

8.00 

3.27 
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4.12  Question  Ten 

What  is  the  cost  per  hour  of  each  person  involved  in  the  meetings? 

Question  ten  provided  the  eost  per  person  to  include  in  the  final  portion  of  the 
cost  of  oversight  estimate  for  meetings  conducted  for  a  milestone.  The  results  of 
question  ten  mirrored  the  results  of  question  seven,  due  to  both  dealing  with  the  cost  of 
personnel  per  hour.  The  results  will  still  be  provided  separately  due  to  future  discussion 
on  the  cost  of  oversight  and  the  analysis  portion  of  the  thesis.  The  estimates  will  be 
provided  in  the  same  format  as  previous  questions. 

4,12,1  Question  Ten  -  Low  Estimate 

The  range  for  the  cost  per  hour  was  changed  very  little  during  the  process.  In  the 
end,  the  range  only  decreased  by  $20.  Members  of  the  panel  made  very  few  changes  to 
their  initial  estimates.  Although  round  three  showed  some  promise  when  a  mode  was 
established,  the  final  results  of  round  four  ended  without  an  established  mode.  Round 
four  ended  with  the  standard  deviation  increasing  from  that  established  in  round  three. 
Members  could  not  agree  on  an  estimate;  therefore,  a  consensus  was  not  met  for  the  low 
estimate.  Results  have  been  listed  in  Table  4.28. 


Table  4.28:  Question  Ten  -  Low  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

$50  to  $125 

1  $60,$100,$50,$125  1 

$  83.75 

$  80.00 

NA 

$ 

34.97 

2 

$70  to $125 

$  93.75 

$  90.00 

NA 

$ 

24.28 

3 

$70  to $125 

1  $70, $100, $100, $125  1 

$  98.75 

$  100.00 

$  100.00 

$ 

22.50 

4 

$70  to  $125 

$  101.25 

$  105.00 

NA 

$ 

23.23 

4.12.2  Question  Ten  -  Average  Estimate 


Initially  there  was  a  very  large  range  amongst  the  panel  members  and  no  mode 


had  been  established.  However,  by  the  third  round  the  panel  members  had  locked  in  their 
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estimates  and  established  a  mode.  The  median  and  mode  were  the  same  and  the  mean 


was  less  than  three  dollars  different  from  the  two.  The  standard  deviation  was  also 
minimized  at  a  low  $5  per  hour.  The  range  had  also  deereased  to  only  $10  and  the  panel 
three  of  the  four  panel  members  had  reaehed  an  agreement  on  the  overall  high  estimate 
eost,  but  that  was  still  one  member’s  estimate  short  of  a  eonsensus.  The  results  ean  be 
seen  below  in  Table  4.29. 


Table  4.29;  Question  Ten  -  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4) 

Mean 

Median 

Mode 

Std  Dev 

1 

$70  to  $200 

1  $150, $200, $70, $140  1 

140.00 

145.00 

NA 

53.54 

2 

$120  to  $200 

152.50 

145.00 

NA 

34.03 

3 

$140  to  $150 

1  $150,$150,$150,$140  1 

147.50 

150.00 

150.00 

5.00 

4 

$140  to  $150 

■ifefaKifefaKifeltlfeiEBM 

147.50 

150.00 

150.00 

5.00 

4,12,3  Question  Ten  -  High  Estimate 


Panel  members  never  really  deeided  on  a  high  estimate.  The  estimates  had  a  wide 


range  in  the  beginning  and  elosed  the  gap  some  by  the  end  of  round  four.  Although  the 


gap  of  the  range  redueed,  there  was  no  real  stability  between  the  panel  members.  There 


were  a  eouple  of  oeeasions  when  a  mode  was  established  but  in  the  end,  there  was  no 


mode  or  eonsensus.  The  standard  deviation  was  also  high  at  the  end  of  round  four.  Panel 


members  were  very  eonhdent  with  their  estimate  early  on  whieh  shows  in  their 


responses.  A  eomplete  listing  of  their  responses  ean  be  found  in  Table  4.30. 


Table  4.30:  Question  Ten  -  High  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3,4) 

Mean 

Median 

Mode 

Std  Dev 

1 

$100  to  $300 

230.00 

260.00 

300.00 

94.52 

2 

$150  to  $230 

1  $300,$300,$150,$230  | 

245.00 

265.00 

300.00 

71.41 

3 

$200  to  $300 

232.50 

214.00 

200.00 

47.17 

4 

$230  to  $300 

1  $300,$200,$230,$240  | 

242.50 

235.00 

NA 

41.93 
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4,12,4  Meeting  Cost  Estimate 

With  all  the  necessary  data  collected,  it  is  now  almost  possible  to  develop  an 
overall  estimate  for  the  cost  of  all  the  meetings  supporting  the  DAB  milestone  review 
process  using  the  previously  mentioned  algorithm.  The  meeting  cost  estimate  will  be 
developed  by  multiplying  the  estimates  from  question  eight;  the  number  of  meetings  held 
from  PEO  preparation  to  DAB  approval,  by  the  estimates  from  question  nine;  the  length 
in  hours  of  each  meeting,  by  the  estimates  from  question  ten;  the  cost  for  one  person 
involved  in  one  meeting.  Using  the  algorithm,  we  are  able  to  develop  a  cost  estimate  for 
one  person  at  all  of  the  meetings  necessary  to  get  through  milestone  approval.  In  order  to 
come  up  with  an  estimate  for  the  cost  of  all  the  meetings,  the  Delphi  panel  was  asked  to 
provide  a  low,  medium  and  high  estimate  for  the  number  of  people  who  attend  the 
meetings.  The  results  can  be  seen  below  in  Table  4.3 1 . 

Table  4.31:  Estimated  Number  of  People  Attending  Meetings  for  One  Milestone 


Member# 

Low 

Med 

High 

1 

5 

12 

25 

2 

6 

15 

40 

3 

10 

20 

30 

4 

5 

10 

25 

Low 

Med 

High 

Mean 

6.50 

14.25 

30.00 

Standard  Dev 

2.38 

4.35 

7.07 

With  all  of  the  necessary  today  now  available,  the  data  was  compiled  and  produced  the 
following  estimates  for  the  cost  of  the  meetings  in  support  of  one  milestone. 
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Table  4.32:  Estimate  of  Meeting  Costs  for  One  Milestone 


Questions  8-11 


MEMBER _  Meeting-LOW  Meeting-AVG  Meeting-HIGH 


1 

$10,500.00 

$360,000.00 

$7,200,000.00 

2 

$7,200.00 

$67,500.00 

$512,000.00 

3 

$300,000.00 

$2,208,000.00 

4 

$19,800.00 

$142,800.00 

$3,072,000.00 

MEAN 

$18,750.00 

$217,575.00 

$3,248,000.00 

STD  DEV 

$13,590.81 

$135,637.47 

$2,841,160.33 

The  estimates  were  ealculated  by  member  for  low,  average  and  high  and  then  a 


mean  and  standard  deviation  was  calculated  for  all  five  members  for  the  low,  average. 


and  high  estimates.  Based  on  the  mean,  the  end  result  of  four  rounds  of  the  Delphi  was  a 


cost  of  meetings  estimate  that  ranges  from  about  $18.75K  to  nearly  $3.25M  for  one 


milestone. 

4.13  Summary  of  Results 

Using  a  simple  algorithm,  we  were  able  to  develop  estimates  for  the  three  major 


oversight  cost  portions  of  a  milestone  review  for  an  IT  or  “virtual”  MDAP.  By  adding 


these  individual  estimates  together,  we  can  arrive  at  an  overall  estimate  for  the  cost  of 


one  milestone  decision  point.  The  results  of  summing  the  estimates  for  travel  cost. 


personnel  cost,  and  meeting  cost  can  be  seen  below  in  Table  4.33. 


Table  4.33:  Estimates  for  Oversight  Cost  of  One  Milestone 


Milestone  Decision  Point  (MDP) 


MEMBER _ MDP  Low _ MDP  Avg _ MDP  High 


1 

$63,540.00 

$787,000.00 

$9,735,000.00 

2 

$56,400.00 

$187,200.00 

$1,073,600.00 

3 

$104,700.00 

$730,000.00 

$4,398,000.00 

4 

$267,480.00 

$1,376,400.00 

$8,016,000.00 

MEAN 

$123,030.00 

$770,150.00 

$5,805,650.00 

STD  DEV 

$98,624.55 

$486,232.36 

$3,860,017.89 
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The  estimates  for  milestone  eosts  were  calculated  in  a  similar  manner  to  the 


estimates  for  travel,  personnel  and  meetings  which  was  by  member  for  low,  average  and 
high  and  then  a  mean  and  standard  deviation  was  calculated  for  all  five  members  for  the 
low,  average,  and  high  estimates.  The  mean  statistic  shows  a  milestone  oversight  cost 
range  from  almost  $123K  to  over  $5.8M.  By  itself,  this  statistic  is  interesting  enough, 
however  more  value  can  be  added  by  comparing  this  range  to  the  range  for  oversight 
costs  developed  by  the  Oversight  and  Review  PAT;  “this  team  came  up  with  an  average 
estimate  of  $10-12  million  for  a  single  milestone  and  an  estimate  of  $40-50  million  for  an 
entire  joint  acquisition  program  in  1994  dollars”  (5:9).  Using  raw  inflation  indices,  the 
ORPAT  team’s  figures  can  be  inflated  to  fiscal  year  2003  dollars  or  the  figures  we 
developed  could  be  brought  back  to  1994  dollars.  We  chose  to  calculate  the  latter  and  the 
results  can  be  seen  below  in  Table  4.34. 


Table  4.34:  FY2003  Milestone  Oversight  Costs  Brought  Back  to  FY1994  Dollars 


Milestone  Decision  Point  (MDP)-Adjusted 


Milestone 

MDP  Low 

MDP  Avg 

MDP  High 

MEAN 

$123,030.00 

$770,150.00 

$5,805,650.00 

3080  Raw  Indice  (1994) 

0.885 

0.885 

0.885 

Adjusted  Mean 

$108,881.55 

$681,582.75 

$5,138,000.25 

As  the  chart  shows,  the  “high”  milestone  estimate  is  nearly  half  of  the  cost 
estimated  by  the  ORPAT  estimate  when  the  dollars  are  the  same.  The  procurement  or 
3080  index  was  used  because  we’re  dealing  with  the  procurement  of  weapons  systems.  If 
the  index  for  wages  was  used,  the  figure  would  be  smaller  since  the  military  wage  index 
is  .750  and  the  civilian  employee  index  is  .738  (38:1). 

After  developing  a  milestone  oversight  cost  estimate,  a  total  program  oversight 
cost  estimate  can  be  created  quite  easily.  Our  methodology  for  developing  a  total 
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program  oversight  cost  was  simply  to  multiply  the  milestone  figures  by  three  to  represent 
the  three  milestones.  The  assumption  here  is  that  each  milestone  costs  relatively  the 
same.  We  decided  to  stick  to  that  assumption  because  a  whole  series  of  research  could  be 
conducted  on  the  cost  difference  from  milestone  to  milestone.  The  results  of  our 
calculation  for  total  milestone  cost  can  be  seen  in  Table  4.35  below. 


Table  4.35:  Estimates  for  Total  Program  Oversight  Costs 


Program 


MEMBER _  Program  Low  Program  Avg  Program  High 


1 

$190,620.00 

$2,361,000.00 

$29,205,000.00 

2 

$169,200.00 

$561,600.00 

$3,220,800.00 

3 

$314,100.00 

$13,194,000.00 

4 

$802,440.00 

$4,129,200.00 

$24,048,000.00 

MEAN 

$369,090.00 

$2,310,450.00 

$17,416,950.00 

STD  DEV 

$295,873.66 

$1,458,697.07 

$11,580,053.66 

Looking  at  the  range  on  the  mean  we  see  a  total  program  oversight  cost  estimate 
from  a  little  over  $369K  to  over  $17.4M.  Again,  these  results  can  be  compared  to  the 
range  developed  by  the  ORPAT  team  of  $40-$50  million.  To  make  the  comparison 
meaningful  we  again  brought  our  figures  back  to  1994  dollars.  The  results  can  be  seen  in 
Table  4.36  below. 

Table  4.36:  FY2003  Total  Oversight  Costs  Brought  Back  to  FY1994  Dollars 


Program 

Program  Low 

Program  Avg 

Program  High 

MEAN 

$369,090.00 

$2,310,450.00 

$17,416,950.00 

3080  Raw  Indice  (1994) 

0.885 

0.885 

0.885 

Adjusted  Mean 

$326,644.65 

$2,044,748.25 

$15,414,000.75 

The  “high”  program  oversight  cost  estimate  of  just  over  $15.4M  is  still  well 
below  the  estimate  of  $40-$50  million  developed  by  the  ORPAT.  The  ORPAT  estimate 
however,  may  be  on  the  high  side  since  back  when  it  was  developed  little  information  on 
the  effectiveness  of  “virtual”  acquisition  programs  were  known. 
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The  goal  of  the  Delphi  Method  was  to  eomplete  at  least  four  rounds  while  trying 
to  reaeh  eonsensus.  The  objeetives  were  elear  for  how  eonsensus  would  be  determined. 
The  rule  was  met  for  all  ten  questions  provided  in  the  survey  and  all  objectives  for  the 
data  collection  portion  were  met. 

Now  that  the  estimates  have  been  provided,  the  information  will  be  placed  in 
statistical  software  as  a  database.  Each  respondent  will  have  their  estimates  entered  for 
each  question.  These  estimates  will  be  compared  with  other  respondents  from  the  theses 
research  conducted  by  Rousseau  (35:1)  and  DeReus  (16:1).  When  comparing  all  of  the 
estimates  together,  an  analysis  of  variance  test  will  be  conducted  by  question,  by  type  of 
regulatory  guidance  policy  programs  typically  fall  under  (i.e.,  NSSAP  03-01,  DoDD 
5000  series,  or  Virtual  oversight).  Once  this  analysis  has  been  completed,  the  results  and 
analysis  will  be  presented  in  chapter  five  to  see  if  there  truly  is  a  difference  in  the  cost  of 
oversight  among  programs. 
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5,0  Analysis 


5.1  Overview 

The  goal  of  Chapter  5  is  to  eompare  the  results  of  the  final  round  of  Delphi 
surveys  for  eaeh  of  the  aequisition  diseiplines  examined.  The  first  seetion  will  eontain  a 
question  by  question  statistieal  eomparison  of  final  responses  submitted  by  the  panel 
members.  Using  the  methodology  deseribed  in  Chapter  3,  a  eomparison  of  the  three 
proeesses  are  eonducted.  Question  one  contains  qualitative  discussions  for  the  cost 
drivers  identified,  but  will  list  all  drivers  to  complete  the  goal  of  research  question  three. 
Questions  two  through  ten  are  quantitatively  compared  with  a  significance  level  of  .05  for 
testing  the  null  hypothesis  of  finding  any  statistical  differences  in  the  mean,  for  the 
forecast  data  collected.  Each  question  will  include  a  discussion  on  where  the  differences 
are  and  discuss  some  of  the  similarities  among  the  different  disciplines,  answering 
research  question  two. 

Recommendations  for  future  research  in  oversight  costs  will  be  discussed  in  the 
final  section.  The  final  section  will  also  provide  any  insights  gained  during  this  research 
which  may  help  make  our  acquisition  process  function  more  efficiently.  Finally,  any 
future  research  efforts  that  could  continue  to  build  on  this  thesis  will  be  provided  to  assist 
in  defining  the  cost  of  oversight  in  MDAPs  in  the  future 

5.2  Question  One 

From  the  Program  Executive  Officer  (PEO)  request  for  a  Defense  Acquisition 

Board  (DAB  to  the  DAB  milestone  approval,  what  are  the  five  major  cost  drivers 

in  the  oversight  process? 

While  the  results  from  the  Space  programs  focuses  on  TDY  expense  and  salaries 
of  contract  employees,  the  results  from  the  IT  and  “Box”  programs  focuses  more  on  the 
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program  and  the  acquisition  processes.  The  cost  drivers  selected  by  the  Box  panel 
members  were  program  driven.  Multi-service  programs,  new  programs,  and  program 
upgrades,  especially  technological  upgrades  were  among  the  top  cost  drivers.  Results 
from  chapter  four  revealed  that  the  IT  community  believed  process  and  requirements 
changes  were  the  main  cost  drivers.  They  also  felt  that  trust  between  OSD  and  program 
leaders  was  an  issue  that  drove  cost  higher.  It  seems  that  both  the  IT  and  Box  programs 
have  issues  at  a  higher  level  than  that  of  Space  programs.  The  cost  drivers  identified  by 
these  two  programs  will  require  some  directive  changes  to  eliminate  most  of  them,  but 
the  drivers  identified  by  the  Space  panel  members  are  at  a  lower  level.  TDYs  can  be 
managed  by  the  PMs  in  most  cases  and  they  usually  have  some  flexibility  on  the  contract 
personnel  hired  to  support  the  program.  Because  both  the  IT  and  Box  programs  are 
required  to  follow  the  guidelines  established  by  the  DoD  5000  series,  they  have  more 
“red  tape”  that  restricts  their  flexibility  to  eliminate  the  unnecessary  obstacles  throughout 
their  MDA  approval  process,  and  the  Space  program  has  been  given  permission  to 
establish  its  own  guidelines  to  follow.  By  doing  so,  they  have  internally  eliminated  the 
“red  tape”.  The  top  five  cost  drivers  selected  by  each  group  of  panel  members  are  shown 
in  table  5.1.  In  the  table,  the  Box  program  is  represented  by  DoDD  5000,  Space  is  simply 
Space,  and  IT  is  C3I. 
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Table  5.1:  Cost  Drivers  for  Oversight  Proeesses 


Drivers  Picked-DoD  5000 

Rank 

Program  is  Multi-Service 

1 

Whether  Completely  new  system  or  just  block  upgrade 

2 

Number  of  Technologies  going  into  the  system 

3 

Number  of  Systems  the  System  must  interact  v«th 

4 

Milestone  B  (requires  most  documents;  30  to  be  generated  for  review) 

5 

1  1 

Drivers  Picked-Soace 

Rank 

Time  away  from  primary  responsibilities  wkiile  supporting  IPA  at  expense  of  rest  of  program 

1 

TOY  from  the  program  office  IPA  or  IPA  folks  to  program  office 

2 

IPA  Personnel  Costs  (Program  Evaluation) 

3 

Salaries  of  IPA  core  members  and  "gray  beard"  members  w/ho  are  not  government  employees 

4 

IPA  Travel/Per  Diem  costs  (Team  and  support  personnel) 

5 

1  1 

Drivers  Picked-C3l 

Rank 

Lack  of  functional  requirements  that  are  clearly  defined  and  understood 

1 

Changing  oversight  requirements. .  .changing  personalities,  policy  etc.--requires  climbing  learning  curve 
again 

2 

Lack  of  established  architectures  and  the  resulting  need  for  unique  C4ISP  efforts 

3 

The  serial  process  of  document  approval  by  the  several  echelons  of  oversight 

4 

Negotiating  view^Doints  of  the  various  stakeholders. .  .acq  strategy  re-do 

4 

5.3  Question  Two 

From  the  PEO  recommendation,  to  the  DAB  approval  of  the  milestone,  use  your 
professional  judgment  and  estimate  how  many  TDYs  are  taken  by  one  person  to 
get  one  program  through  one  Milestone. 

When  comparing  question  two,  there  were  statistical  differences  for  all  three 
oversight  processes  when  comparing  the  Box  and  Space  programs  with  the  IT  programs 
except  for  one  (high  estimate  of  the  Box  and  IT  programs).  Table  5.2  shows  that  at  all 
levels  of  the  forecasts,  none  of  the  estimates  for  the  Space  program  were  statistically 
similar  to  IT  oversight.  However,  the  shaded  area  shows  that  the  high  estimate  of  the 
Box  program  is  statistically  similar  to  the  high  estimate  of  the  IT  program.  Because  both 
programs  follow  the  same  guidelines,  the  worst  case  scenarios  should  be  somewhat 
similar  which  may  be  the  reason  the  high  estimates  of  the  two  programs  are  similar. 
Although  one  would  expect  the  IT  programs  would  have  the  least  amount  of  TDYs 
throughout  the  approval  process  for  each  milestone,  the  data  shows  that  the  Space 
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program  actually  goes  on  fewer  TDYs  than  both  the  IT  and  Box  programs.  This  is  ironie 
beeause  under  the  IT  proeess,  program  information  is  aoeessible  to  all  interested  parties. 

Table  5.2:  ANOVA  for  Question  Two 


Question  2 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

C3I  vs  Space 

0.0004 

0.0038 

0.0075 

C3I  vs  5000 

0.008 

0.0317 

0.1766 

5.4  Question  Three 

Estimate  how  many  people  normally  go  TDY  throughout  the  Milestone  Decision 
process. 

In  question  three,  there  were  not  any  significant  statistical  differences  when 
looking  for  the  number  of  people  that  aetually  go  TDY  in  the  MDA/KDP  proeess. 
Aeeording  to  the  information  provided  in  table  5.3,  there  are  no  statistieal  differences 
between  the  different  oversight  proeesses  in  the  number  of  people  going  TDY.  Although 
every  member  in  the  IT  approval  proeess  has  aeeess  to  program  data,  there  is  still  a 
relatively  higher  number  of  people  traveling  in  support  of  their  programs  during  the 
MDA  proeess.  Therefore,  eontrary  to  what  one  might  expect,  the  IT  proeess  has  only  a 
slightly  lower  number  of  people  going  TDY  than  that  of  either  the  Box  or  Spaee 


programs. 


Table  5.3:  ANOVA  for  Question  Three 


Question  3 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

C3I  vs  Space 

0.0811 

0.1071 

0.1345 

C3I  vs  5000 

0.2769 

0.2335 

0.1612 
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5.5  Question  Four 

What  is  your  estimate  of  the  cost  for  each  person  on  each  TDY? 

The  results  of  question  four  when  eompared  to  the  other  two  program  types 
varied.  The  low  estimate  revealed  that  there  were  statistieal  differenees  between  Spaee 
and  IT;  however,  there  were  no  statistical  differences  at  the  average  and  high  estimates. 
On  the  other  hand,  there  were  no  statistical  differences  with  the  Box  program  at  the  low 
estimate  level,  but  there  were  statistical  differences  at  both  the  average  and  high  estimate 
levels.  When  comparing  cost  per  person  for  the  IT  programs’  personnel  to  that  of  the 
Box  and  Space  programs,  the  comparisons  are  total  opposites.  Differences  amongst  the 
different  program  types  are  found  in  table  5.4. 


Table  5.4:  ANOVA  for  Question  Four 


Question  4 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

C3I  vs  Space 

0.0008 

0.0582 

0.1143 

C3I  vs  5000 

0.4071 

0.0123 

0.0017 

5.6  Question  Five 

Estimate  how  many  hours  are  spent  on  support  for  the  DAB  approval  process  per 
person,  not  including  TDY  travel  time,  but  actual  job  performance  while  TDY  or 
at  home  base.  (Slide  prep,  meeting  prep,  etc) 

Question  five  results  are  located  in  Table  5.5  and  it  shows  that  there  were 
statistical  differences  between  IT  and  the  other  two  programs  at  each  estimate  level. 

When  comparing  IT  estimates  with  the  Space  estimates,  the  preparation  hours  were  lower 
in  the  IT  programs  at  the  lower,  average,  and  higher  estimate  levels.  Information 
accessibility  should  be  given  credit  for  the  differences.  One  would  believe  that  the 
availability  of  program  data  to  all  concerned  parties  help  eliminate  the  need  for  numerous 
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hours  of  meeting  preparation.  Any  information  that  may  be  needed  is  already  available. 
PMs  would  only  have  to  prepare  for  the  possible  questions  about  the  data  that  may  need 
to  be  answered  during  the  meeting. 

The  IT  program’s  estimates  were  also  lower  than  that  of  the  Box  programs.  Onee 
again  the  aeeessibility  of  the  program  data  in  the  IT  programs  made  the  differenee.  The 
Box  programs  are  usually  overloaded  with  questions  about  their  programs.  Beeause 
miniseule  details  of  the  program  data  are  held  at  the  program  offiee  levels,  PMs  are 
required  to  flow  that  data  to  the  upper  levels  of  management.  Therefore,  they  must  spend 
more  time  preparing  for  meetings  in  order  to  be  able  to  answer  the  questions  that  may 
eome  up  during  the  meetings.  The  number  of  meetings  required  between  each  level  of 
approval  also  contribute  to  the  larger  amount  of  time  required  for  the  Box  programs. 

This  is  a  significant  finding  because  it  shows  potential  cost  reductions  a  virtual  process 
may  have  if  adopted  by  other  programs. 


Table  5.5;  ANOVA  for  Question  Five 


Question  5 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

C3I  vs  Space 

0.0019 

0.0012 

0.0007 

C3I  vs  5000 

0.0039 

0.0003 

0.0002 

5.7  Question  Six 

Estimate  how  many  people  are  normally  involved  with  the  preparation  process. 

Once  again  the  statistical  data  reveals  that  the  IT  and  Space  programs  are 
statistically  different  in  all  three  estimates.  Again,  it  is  assumed  that  the  virtual  process 
of  data  availability  used  in  the  IT  programs  is  the  main  reason  for  this  difference.  It  is 
assumed  that  when  data  is  available  to  everyone  in  the  decision  making  process,  there  are 
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fewer  people  required  to  prepare  that  decision  maker  for  a  meeting  because  the 
information  needed  is  at  his  or  her  fingertips.  Based  on  the  raw  data,  the  number  of 
people  normally  involved  in  the  IT  preparation  process  is  lower  than  the  Space  and  Box 
programs.  Although  the  number  of  people  involved  in  the  preparation  process  in  the  IT 
program  is  lower  than  the  Box  programs,  statistically,  there  are  no  differences  between 
the  two  programs.  Although  it  seems  impossible,  the  two  programs  are  required  to 
follow  the  same  DoD  5000  series’  rules  and  regulations;  therefore,  the  two  programs 
should  not  be  statistically  different.  The  number  of  meetings  required  and  the  essential 
personnel  are  set  by  regulation  which  may  explain  the  statistical  similarities  of  the 
programs.  The  statistical  comparisons  of  the  programs  are  shown  below  in  table  5.6. 


Table  5.6;  ANOVA  for  Question  Six 


Question  6 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

C3I  vs  Space 

0.0001 

0.002 

0.0001 

C3I  vs  5000 

0.1906 

0.1331 

0.0587 

5.8  Question  Seven 

Estimate  the  cost  per  hour  for  each  person  involved  in  the  process. 

For  the  most  part,  there  is  a  significant  difference  between  IT  and  Space  and  IT 
and  Box  programs.  However,  there  is  no  statistical  difference  between  the  IT  and  Space 
programs  for  the  high  estimates.  The  raw  data  shows  that  estimates  for  the  lower  and 
average  cost  were  higher  for  IT  programs,  but  the  high  estimates  of  the  two  programs 
were  somewhat  closer  in  terms  of  dollars.  However,  the  results  for  the  IT  program  are 
significantly  higher  in  all  areas  than  the  Box  estimates.  These  differences  may  be  due  to 
the  pay  grades  involved.  While  Box  programs  generally  have  lower  grades  involved  in 
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the  preparation  proeess  to  get  the  PMs  prepared  for  meetings,  IT  managers  uses  the 
virtual  network  to  get  the  neeessary  information.  And  although  less  hours  are  spent 
preparing  for  meetings,  the  eost  per  hour  is  higher;  therefore,  the  eost  per  hour  will  be 
signifieantly  higher  for  the  IT  program.  The  statistieal  eomparisons  of  the  different 
programs  ean  be  seen  below  in  table  5.7. 


Table  5.7:  ANOVA  for  Question  Seven 


Question  7 

p-Values  (.05  significance  ievei) 

COMPARISON 

LOW 

AVG 

HIGH 

C3I  vs  Space 

0.003 

0.0001 

0.2178 

C3I  vs  5000 

0.0012 

0.0001 

0.0004 

5.9  Question  Eight 

Estimate  how  many  meetings  are  normally  held  from  the  PEO  preparation, 
through  DAB  approval.  (This  includes  meetings  TDY or  TDY prep  meetings). 

Of  all  the  eomparisons,  there  was  a  single  statistieal  significanee  found.  When 

eomparing  the  means  in  the  ANOVA  test,  the  low  estimates  for  the  Spaee  and  IT 

programs  there  was  a  signifieant  statistical  difference.  Although  the  other  comparisons 

did  not  show  any  differences  at  the  .05  significance  level,  there  were  still  some  notable 

differences  when  reviewing  the  main  data  collected.  Both  the  IT  and  Space  programs 

had  a  low  number  of  meetings  while  the  Box  programs  continued  to  show  a  large  number 

of  meetings  held  during  the  MDA  approval  process.  One  might  conclude  that  the 

waivers  to  the  DoD  5000  series’  procedures  obtained  by  these  two  programs  allowed 

them  to  streamline  the  meetings  being  held  to  obtain  MDA  and  KDP  approval.  However, 

based  on  the  results,  there  are  no  statistical  differences  between  the  number  of  meetings 

held  in  IT  programs  and  either  Space  or  Box  programs.  The  differences  will  appear  when 
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dollar  values  are  placed  on  those  meetings  held  by  each  program.  Comparisons  of  the 
three  program’s  number  of  meetings  held  are  shown  in  table  5.8  below. 


Table  5.8:  ANOVA  for  Question  Eight 


Question  8 

p-Values  (.05  significance  ievei) 

COMPARISON 

LOW 

AVG 

HIGH 

C3I  vs  Space 

0.0372 

0.1248 

0.1306 

C3I  vs  5000 

0.1129 

0.2234 

0.2686 

5.10  Question  Nine 

What  do  you  estimate  as  the  length,  in  hours,  for  each  meeting? 

Comparisons  for  the  estimated  length  of  meetings  between  IT  and  Space  as  well 
as  between  IT  and  Box  programs  are  shown  in  table  5.9.  The  results  shows  there  are  no 
statistical  differences  in  either  comparison.  As  a  matter  of  fact,  the  mean  of  the  IT  and 
Space  programs  are  identical.  Although  the  raw  data  for  total  time  required  for  meetings 
show  differences  amongst  the  programs,  the  statistical  data  of  the  mean  shows  there  is 
not  enough  evidence  to  prove  a  statistical  difference. 


Table  5.9:  ANOVA  for  Question  Nine 


Question  9 

p-Values  (.05  significance  ievei) 

COMPARISON 

LOW 

AVG 

HIGH 

C3I  vs  Space 

0.0601 

1.00 

0.6963 

C3I  vs  5000 

0.2977 

0.3489 

0.3326 

5.11  Question  Ten 

What  is  the  cost  per  hour  of  each  person  involved  in  the  meetings? 

Once  again,  there  appear  to  be  statistical  differences  between  IT  programs  and 
that  of  the  Space  and  Box  programs.  Only  the  high  estimates  of  IT  and  Space  programs 
show  no  statistical  differences.  The  meeting  cost  per  hour  for  the  IT  programs  are 
significantly  higher  than  both  the  Space  and  Box  programs.  In  some  instances,  the  IT 
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cost  nearly  doubles  and  sometimes  triples  the  eost  of  the  other  programs.  However,  at 
the  high  estimate  level  there  is  not  enough  statistieal  evidenee  to  prove  a  differenee 
between  the  IT  and  Spaee  programs.  The  pay  grades  of  the  individuals  attending  these 
meetings  must  again  be  attributed  for  the  signifieant  differenees  between  the  programs. 
Since  most  of  the  pre -meetings  are  eliminated  from  use  of  the  real-time  virtual  database, 
high  level  personnel  meet  to  make  the  deeisions  neeessary  for  program  progression. 
Although  the  raw  data  is  not  shown,  table  5.10  shows  the  statistieal  data  for  the  means. 

Table  5.10;  ANOVA  for  Question  Ten 


Question  10 

p-Values  (.05  significance  ievel) 

COMPARISON 

LOW 

AVG 

HIGH 

C3I  vs  Space 

0.0028 

0.0001 

0.1008 

C3I  vs  5000 

0.0026 

0.0001 

0.0002 

5.12  Summary  of  Results 

Overall,  the  three  researeh  questions  that  were  stated  as  goals  for  this  thesis  have 
been  answered.  The  total  cost  of  oversight  has  been  ealculated  as  an  estimate  for  the 
MDA  proeess  under  the  RIT  Pilot  Study  proeess,  the  top  five  eost  drivers  have  been 
identified,  and  finally,  when  compared  to  the  other  proeesses,  the  researeh  question 
dealing  with  any  statistieal  differenees  in  the  eost  of  oversight  between  the  different 
oversight  proeesses  has  been  answered  for  all  but  one  eombination;  total  eost 
eomparison. 

After  getting  answers  to  the  other  questions,  a  final  eomparison  for  the  total  cost 
of  oversight  was  taken  to  determine  if  there  were  any  statistieal  differenees  between  the 
three  program  types.  Given  our  assumptions  from  chapter  one,  results  of  the  analysis 
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show  no  statistical  differences  for  the  eost  of  oversight  between  the  programs.  Results  of 
the  analysis  are  shown  in  table  5.11  below. 


Table  5.11:  ANOVA  for  Total  Cost  Comparison 


Total  Cost 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  031 

0.1625 

0.1932 

0.5188 

Space  vs  5000 

0.1393 

0.117 

0.1629 

5000  vs  C3I 

0.1655 

0.1557 

0.343 

Although  no  statistieal  differenees  were  shown  when  the  data  was  analyzed,  the 


ranges  for  total  cost  were  very  different.  The  total  range  of  eost,  estimated  by  the  panel 


members  for  the  three  programs,  was  as  low  as  $82K  to  as  high  as  $94M.  A  eomplete 


look  at  the  ranges  ean  be  found  in  table  5.12. 


Table  5.12:  Total  Cost  Ranges  by  Oversight  Proeess  within  Range 


Process 

Low  Range 

Average  Range 

High  Range 

C3I 

$201 ,000  to  $861 ,840 

$696,600  to  $4,471 ,920 

$3,028,800  to  $52,605,000 

Space 

$82,434  to  $222,000 

$1,271 ,250  to  $1,779, 187 

$11, 443,500  to  $26,01 9,000 

5000 

$41 7,600  to  $8,581 ,680 

$3,219,960  to  $27,931,500 

$11, 838,240  to  94,968,000 

With  sueh  a  range  of  eost,  it  is  inevitable  that  eaeh  proeess  has  its  own  cost  savings 


potential.  Table  5.13  shows  the  means  of  the  total  program  eost.  As  you  ean  see  the 


Space  program  has  the  lowest  average  for  the  total  program  cost;  but  in  the  following 


seetion,  reeommendations  to  reduee  eost  even  lower  will  be  made. 


Table  5.13:  Average  (Mean)  Total  Cost  by  Program 


Process 

Low  Range 

Average  Range 

High  Range 

C3I 

$399,735 

$2,777,505 

$23,679,750 

Space 

$146,379.75 

$1,534,359.38 

$15,979,687.50 

5000 

$3,758,590.08 

$13,417,272 

$44,211,936 
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5.13  Recommendations 


After  calculating  the  total  costs  for  the  IT  process  and  comparing  those  costs  with 
the  Space  and  Box  programs,  perhaps  there  is  a  way  to  reduce  the  overall  program  cost 
for  all  programs  if  processes  are  taken  from  the  programs  with  the  least  amount  of 
oversight  cost.  The  results  show  that  the  Space  program  has  the  lowest  TDY  and  meeting 
cost  and  the  IT  program  has  the  lowest  preparation  cost.  By  combining  some  of  the 
Space  and  IT  procedures,  there  may  be  evidence  that  the  total  oversight  cost  can  be 
reduced.  Therefore,  I  would  recommend  using  the  IPA  process  but  the  virtual 
accessibility  of  the  program  data  should  also  be  incorporated  into  the  process.  This 
process  should  produce  the  least  amount  of  cost  based  on  the  comparisons  made  from  this 
study  and  the  studies  conducted  by  DeReus  (16:1)  and  Rousseau  (35:1).  If  these  changes 
are  made  for  all  programs  in  the  acquisition  process,  there  is  much  potential  for  the 
overall  program  cost  to  decrease.  Of  the  three  programs,  the  mean  total  program  cost  for 
the  Space  program  was  lower  in  each  of  the  categories,  low,  average  and  high  cost 
estimates.  However,  if  you  used  the  overall  meeting  and  travel  cost  from  the  original 
Space  program  numbers  and  add  that  number  to  the  personnel  cost  from  the  IT  program, 
DoD  could  realize  some  savings  in  oversight  cost.  Although  more  research  in  this  area 
would  need  to  be  done  to  validate  this  proposal,  by  implementing  a  virtual  database  into 
the  Space  programs’  acquisition  process  the  total  program  cost  for  program  oversight  can 
be  reduced  by  as  little  as  $55K  to  more  than  $4.5M.  The  funds  can  be  realized  either 
directly  or  indirectly  by  saving  man-hours  or  actual  bottom-line  budget  savings.  Either 
way,  some  sort  of  live  program  test  or  feasibility  study  should  be  performed.  An 
example  of  the  potential  oversight  cost  savings  are  shown  in  table  5.14. 
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Table  5.14:  Potential  Cost  Reduetions  with  IPA  +  Virtual  Database 


Process 

Low  Range 

Average  Range 

High  Range 

Space 

$146,379.75 

$1,534,359.38 

$15,979,687.50 

SpaceA^irtual 

$91,404.75 

$865,771.88 

$11,475,450.00 

Potential  Savings 

$54,975.00 

$668,587.50 

$4,504,237.50 

5.14  Follow-on  Possibilities 


This  researeh  was  just  a  beginning.  This  study  produeed  a  baseline  to  ealeulating 
the  eost  of  oversight  for  a  major  defense  aequisition  program,  but  there  is  a  lot  of 
potential  for  further  researeh  in  this  area.  A  study  of  the  Ballistic  Missile  Defense 
Agency’s  acquisition  process  in  comparison  of  the  IT,  Space,  and  Box  programs  could  be 
conducted.  Cost  oversight  below  the  PM  level  can  also  be  researched  to  determine  if 
there  is  potential  to  save  additional  funds  by  streamlining  the  oversight.  Because  this 
study  focused  only  on  Air  Force  programs,  further  studies  can  be  conducted  for  the  cost 
of  oversight  of  the  sister  services  (i.e.  Army,  Navy,  and  Marines).  A  study  that  compares 
the  oversight  costs  of  the  Air  Force  and  its  sister  services  is  another  research  project. 
These  are  just  a  few  of  the  endless  possibilities  of  research  in  this  area.  This  research  has 
provided  a  basic  way  to  identify  oversight  cost  within  MDAPs. 
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followed.  These  oversight  forums  track  a  program’s  progress  throughout  the  milestone  process  and  identified  programs  in  trouble.  Changes  to  the 
DoD  5000  have  been  made  because  of  political  and  economical  changes  over  the  years.  However,  few  studies  have  been  conducted  to  estimate  the 
cost  of  oversight,  and  no  one  knows,  for  sure,  how  much  this  oversight  process  costs  individual  programs  throughout  its  life  cycle.  There  are 
several  oversight  processes  being  used  today,  but  research  has  been  done  to  determine  if  one  form  is  statistically  better  than  the  other.  Nor  have 
studies  been  done  to  determine  the  cost  drivers  for  oversight. 

This  thesis  will  provide  a  foundation  and  potential  cost  saving  recommendations  that  would  benefit  the  Department  of  Defense  in  most  of 
the  acquisition  programs  it  monitors.  The  cost  of  oversight  will  be  forecasted  based  on  a  panel  of  experts  in  the  field,  using  the  Delphi 
Methodology.  These  costs  will  then  compare  with  the  oversight  processes  for  the  Department  of  Defense  Directive  5000  and  the  new  National 
Security  Space  Policy  for  acquisitions.  Total  costs  for  the  three  oversight  processes  will  then  provide  insight  on  where  cost  benefits  appear,  based 
on  collected  data.  A  future  track  for  the  next  generation  of  oversight  processes  will  develop  from  the  recommendations. 
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